Three-dimensional displays using electromagnetic field computations

ABSTRACT

Methods, apparatus, devices, and systems for three-dimensional (3D) displaying objects are provided. In one aspect, a method includes obtaining data including respective primitive data for primitives corresponding to an object, determining an electromagnetic (EM) field contribution to each element of a display for each of the primitives by calculating an EM field propagation from the primitive to the element, generating a sum of the EM field contributions from the primitives for each of the elements, transmitting to each of the elements a respective control signal for modulating at least one property of the element based on the sum of the EM field contributions, and transmitting a timing control signal to an illuminator to activate the illuminator to illuminate light on the display, such that the light is caused by the modulated elements of the display to form a volumetric light field corresponding to the object.

INCORPORATION BY REFERENCE

The present application is a continuation of, and claims benefit under35 USC § 120 to, international application PCT/US2019/013857, filed Jan.16, 2019, and entitled “Three-Dimensional Displays Using ElectromagneticField Computations,” which claims benefit under 35 U.S.C. § 119 to U.S.Ser. No. 62/618,054, filed Jan. 16, 2018, and entitled“Three-Dimensional Displays Using Electromagnetic Field Computations.”The entire contents of both applications are incorporated by referenceherein.

TECHNICAL FIELD

This disclosure relates to three-dimensional (3D) displays, and moreparticularly to 3D displays using computation technology.

BACKGROUND

Advances in traditional two-dimensional (2D) projection and 3D renderinghave led to new approaches for 3D displays, including numerous hybridtechniques that mix head and eye tracking with conventional displaydevices for virtual reality (VR), augmented reality (AR), and mixedreality (MR). These techniques attempt to replicate an experience ofholographic imagery, combined with tracking and measurement-basedcalculations, to simulate stereo or in-eye light field that can berepresented by an actual hologram.

SUMMARY

The present disclosure describes methods, apparatus, devices, andsystems for using electromagnetic (EM) field computations forthree-dimensional (3D) displays.

The present disclosure provides technology that can overcome limitationspresent in known technologies. As an example, the technology disclosedherein can be implemented without the use of cumbersome wearabledevices, such as “3D glasses.” As another example, the technologydisclosed herein can optionally be implemented without being limited byaccuracy tracking mechanisms, the quality of the display devices,relatively long processing times and/or relatively high computationaldemands, and/or by an inability to display objects to multiple viewerssimultaneously. As a further example, the technology can be implementedwithout specialized tools and software to develop contents that extendabove and beyond the tools and software used in conventional 3D contentcreation. Various embodiments may exhibit one or more of the foregoingadvantages. For example, certain implementations of the presentdisclosure can produce real-time, full color, genuine 3D images thatappear to be real 3D objects in the world and can be viewed withoutencumbrances by multiple viewers simultaneously from different points.

One aspect of the present disclosure features a method including: foreach of a plurality of primitives corresponding to an object in athree-dimensional (3D) space, determining an electromagnetic (EM) fieldcontribution to each of a plurality of elements of a display bycomputing, in a 3D coordinate system, EM field propagation from theprimitive to the element; and for each of the plurality of elements,generating a sum of the EM field contributions from the plurality ofprimitives to the element.

The EM field contribution can include at least one of a phasecontribution or an amplitude contribution. The primitives can include atleast one of a point primitive, a line primitive, or a polygonprimitive. The primitives can include a line primitive including atleast one of a gradient color, a textured color, or any surface shadingeffect. The primitives can also include a polygon primitive including atleast one of a gradient color, a textured color, or any surface shadingeffect. The plurality of primitives can be indexed in a particularorder.

In some implementations, the method further includes obtainingrespective primitive data for each of the plurality of primitives. Therespective primitive data of each of the plurality of primitives caninclude respective color information of the primitive, and thedetermined EM field contributions for each of the elements includeinformation corresponding to the respective color information of theprimitives. The color information can include at least one of a texturedcolor or a gradient color. The respective primitive data of each of theplurality of primitives can include texture information of theprimitive. The respective primitive data of each of the plurality ofprimitives can include shading information on one or more surfaces ofthe primitive. The shading information can include a modulation on atleast one of color or brightness on the one or more surfaces of theprimitive.

In some implementations, the respective primitive data of each of theplurality of primitives includes respective coordinate information ofthe primitive in the 3D coordinate system. Respective coordinateinformation of each of the plurality of elements in the 3D coordinatesystem can be determined based on the respective coordinate informationof the plurality of primitives in the 3D coordinate system. Therespective coordinate information of each of the elements can correspondto a logical memory address for the element stored in a memory.

Determining the EM field contribution to each of the plurality ofelements for each of the plurality of primitives can includedetermining, in the 3D coordinate system, at least one distance betweenthe element and the primitive based on the respective coordinateinformation of the element and the respective coordinate information ofthe primitive. In some examples, determining the EM field contributionto each of the plurality of elements for each of the plurality ofprimitives includes: determining a first distance between a firstprimitive of the plurality of primitives and a first element of theplurality of elements based on the respective coordinate information ofthe first primitive and the respective coordinate information of thefirst element; and determining a second distance between the firstprimitive and a second element of the plurality of elements based on thefirst distance and a distance between the first element and the secondelement. The distance between the first element and the second elementcan be predetermined based on a pitch of the plurality of elements ofthe display.

In some examples, at least one of the plurality of primitives is a lineprimitive including first and second endpoints, and determining at leastone distance between the element and the primitive includes: determininga first distance between the element and the first endpoint of the lineprimitive; and determining a second distance between the element and thesecond point of the line primitive. In some examples, at least one ofthe plurality of primitives is a triangle primitive including first,second, and third endpoints, and determining at least one distancebetween the element and the primitive includes: determining a firstdistance between the element and the first endpoint of the triangleprimitive; determining a second distance between the element and thesecond point of the triangle primitive; and determining a third distancebetween the element and the third point of the triangle primitive.

In some implementations, determining the EM field contribution to eachof the plurality of elements for each of the plurality of primitivesincludes determining the EM field contribution to the element from theprimitive based on a predetermined expression for the primitive and theat least one distance. In some cases, the predetermined expression isdetermined by analytically calculating the EM field propagation from theprimitive to the element. In some cases, the predetermined expression isdetermined by solving Maxwell's equations. The Maxwell's equations canbe solved by providing a boundary condition defined at a surface of thedisplay. The boundary condition can include a Dirichlet boundarycondition or a Cauchy boundary condition. The plurality of primitivesand the plurality of elements can be in the 3D space, and a surface ofthe display can form a portion of a boundary surface of the 3D space. Insome cases, the predetermined expression includes at least one offunctions including a sine function, a cosine function, or anexponential function, and determining the EM field contribution includesidentifying a value of the at least one of the functions in a tablestored in a memory.

In some implementations, determining the EM field contribution to eachof the plurality of elements for each of the plurality of primitives andgenerating the sum of the field contributions for each of the pluralityof elements includes: determining first EM field contributions from theplurality of primitives to a first element of the plurality of elementsand summing the first EM field contributions for the first element; anddetermining second EM field contributions from the plurality ofprimitives to a second element of the plurality of elements and summingthe second EM field contributions for the second element. Determiningthe first EM field contributions from the plurality of primitives to thefirst element can include: determining an EM field contribution from afirst primitive of the plurality of primitives to the first element inparallel with determining an EM field contribution from a secondprimitive of the plurality of primitives to the first element.

In some implementations, determining the EM field contribution to eachof the plurality of elements for each of the plurality of primitivesincludes: determining first respective EM field contributions from afirst primitive of the plurality of primitives to each of the pluralityof elements; and determining second respective EM field contributionsfrom a second primitive of the plurality of primitives to each of theplurality of elements, and generating the sum of the field contributionsfor each of the plurality of elements can include: accumulating the EMfield contributions for the element by adding the second respective EMfield contribution to the first respective EM field contribution for theelement. Determining the first respective EM field contributions fromthe first primitive to each of the plurality of elements can beperformed in parallel with determining the second respective EM fieldcontributions from the second primitive to each of the plurality ofelements.

Determining the EM field contribution to each of the plurality ofelements for each of the plurality of primitives can include:determining a first EM field contribution from a first primitive of theplurality of primitives to a first element of the plurality of elementsin parallel with determining a second EM field contribution from asecond primitive of the plurality of primitives to the first element.

In some implementations, the method further includes: for each of theplurality of elements, generating a respective control signal based onthe sum of the EM field contributions from the plurality of primitivesto the element, the respective control signal being for modulating atleast one property of the element based on the sum of the EM fieldcontributions from the plurality of primitives to the element. The atleast one property of the element can include at least one of arefractive index, an amplitude index, a birefringence, or a retardance.The respective control signal can include an electrical signal, anoptical signal, a magnetic signal, or an acoustic signal. In some cases,the method further includes: multiplying a scale factor to the sum ofthe field contributions for each of the elements to obtain a scaled sumof the field contributions, and the respective control signal isgenerated based on the scaled sum of the field contributions for theelement. In some cases, the method further includes: normalizing the sumof the field contributions for each of the elements, and the respectivecontrol signal is based on the normalized sum of the field contributionsfor the element. The method can also include: transmitting therespective control signal to the element.

In some implementations, the method further includes: transmitting acontrol signal to an illuminator, the control signal indicating to turnon the illuminator such that the illuminator emits light on the display.The control signal can be transmitted in response to determining acompletion of obtaining the sum of the field contributions for each ofthe plurality of elements. The modulated elements of the display cancause the light to propagate in different directions to form avolumetric light field corresponding to the object in the 3D space. Thevolumetric light field can correspond to a solution of Maxwell'sequations with a boundary condition defined by the modulated elements ofthe display. The light can include a white light, and the display can beconfigured to diffract the white light into light with different colors.

In some implementations, the method further includes representing valuesusing fixed point number representations during calculation. Each of thevalues can be represented as integers with an implicit scale factor.

In some implementations, the method further includes performing amathematical function using fixed point number representations. Themathematical function can include at least one of sine, cosine, and arctangent. Performing the mathematical function can include receiving anexpression in a first fixed point format, and outputting a value at asecond fixed point format that has a level of accuracy different fromthat of the first fixed point format. Performing the mathematicalfunction can include looking up a table for calculation of themathematical function, wherein the table includes at least one of afully enumerated look-up table, an interpolated table, a semi-tablebased polynomial functions, and a semi-table based on full minimaxpolynomials. Performing the mathematical function can include applying aspecialized range reduction for an input. Performing the mathematicalfunction can include transforming a trigonometric calculation from arange [−π, π] into a signed 2's compliment representation in a range[−1,1].

Another aspect of the present disclosure features a method thatincludes: obtaining respective primitive data of a plurality ofprimitives corresponding to an object in a three-dimensional (3D) space;calculating first respective electromagnetic (EM) field contributionsfrom a first primitive of the plurality of primitives to each of aplurality of elements of a display; and calculating second respective EMfield contributions from a second primitive of the plurality ofprimitives to each of the plurality of elements of the display.Calculating the first respective EM field contributions from the firstprimitive is at least partially in parallel with calculating the secondrespective EM field contributions from the second primitive.

In some implementations, calculating a first EM field contribution fromthe first primitive to a first element of the plurality of elements isin parallel with calculating a second EM field contribution from asecond primitive of the plurality of primitives to the first element.The method can include calculating respective EM field contributionsfrom each of the plurality of primitives to each of the plurality ofelements. The calculation of the respective EM field contributions canbe without at least one of: expanding geometry of the object into theplurality of elements; applying visibility tests before packingwavefronts; and decision making or communication between parallelcalculations for different primitives. The calculation of the respectiveEM field contributions can be configured to cause at least one of:tuning parallel calculations for different primitives to speed, cost,size or energy optimization; reducing latency between initiating a drawand a result being ready for display; increasing accuracy using fixedpoint number representations; and optimizing computation speed byoptimizing mathematical functions.

In some implementations, the method further includes representing valuesusing fixed point number representations during calculation.Representing the values using the fixed point number representations canwithout at least one of: denormal floats for gradual underflow; handlingNaN results from operations including division by zero; alteringfloating point rounding modes; and raising floating point exceptions toan operating system.

In some implementations, the method further includes, for each of theplurality of elements, accumulating EM field contributions for theelement by adding the second respective EM field contribution for theelement to the first respective EM field contribution for the element.

In some implementations, the method further includes, for each of theplurality of elements, generating a respective control signal based on asum of the EM field contributions from the plurality of primitives tothe element, wherein the respective control signal is for modulating atleast one property of the element based on the sum of the EM fieldcontributions from the plurality of primitives to the element.

In some implementations, the method further includes scaling a firstprimitive adjacent to a second primitive by a predetermined factor suchthat a reconstruction of the first primitive does not overlap with areconstruction of the second primitive. The predetermined factor can bedetermined at least partially based on a resolution of the display. Themethod can further include: obtaining respective primitive data for eachof the plurality of primitives, wherein the respective primitive data ofeach of the plurality of primitives comprises respective coordinateinformation of the primitive in the 3D coordinate system; anddetermining new respective coordinate information of the first primitivebased on the respective coordinate information of the first primitiveand the predetermined factor. The method can further include determiningan EM field contribution from the first primitive to each of theplurality of elements based on the new respective coordinate informationof the first primitive. The method can further include scaling thesecond primitive by the predetermined factor. The first primitive andthe second primitive can share a common part, wherein scaling the firstprimitive comprises scaling the common part of the first primitive.Scaling the first primitive can include scaling the first primitive in apredetermined direction.

Another aspect of the present disclosure features a method thatincludes: obtaining respective primitive data of a plurality ofprimitives corresponding to an object in a three-dimensional (3D) space;scaling a first primitive adjacent to a second primitive by apredetermined factor using the respective primitive data for the firstprimitive and the second primitive; and updating the respectiveprimitive data for the first primitive based on a result of the scaling.

In some implementations, the respective primitive data of each of theplurality of primitives include respective coordinate information of theprimitive in a 3D coordinate system, and updating the respectiveprimitive data includes determining new respective coordinateinformation of the first primitive based on the respective coordinateinformation of the first primitive and the predetermined factor.

In some implementations, the predetermined factor is determined suchthat a reconstruction of the first primitive does not overlap with areconstruction of the second primitive in the 3D space.

In some implementations, the scaling is performed such that a gapbetween reconstruction of the first primitive and the second primitivein the 3D space is big enough to separate the first and secondprimitives to minimize an overlapping effect and small enough to makethe reconstruction appear seamless.

In some implementations, the predetermined factor is determined at leastpartially based on a resolution of the display.

In some implementations, the method further includes storing the updatedprimitive data for the first primitive in a buffer.

In some implementations, the scaling is performed during a renderingprocess of the object for obtaining the respective primitive data of theplurality of primitives.

In some implementations, the method further includes transmittingupdated primitive data for the plurality of primitives to a controller,wherein the controller is configured to determining respectiveelectromagnetic (EM) field contributions from each of the plurality ofprimitives to each of a plurality of elements of a display based on theupdated primitive data for the plurality of primitives.

In some implementations, the method further includes determining an EMfield contribution from the first primitive to each of a plurality ofelements of a display based on the updated primitive data of the firstprimitive.

In some implementations, the method further includes scaling the secondprimitive by the predetermined factor.

In some implementations, the first primitive and the second primitiveshare a common part, and scaling the first primitive comprises scalingthe common part of the first primitive.

In some implementations, scaling the first primitive includes scalingthe first primitive in a predetermined direction.

In some implementations, scaling the first primitive includes scaling afirst part of the first primitive by a first predetermined factor, andscaling a second part of the second primitive by a second predeterminedfactor, where the first predetermined factor is different from thesecond predetermined factor.

Another aspect of the present disclosure features a method thatincludes: obtaining a plurality of discrete cosine transform (DCT)weights of an image to be mapped on a specified surface of a particularprimitive of a plurality of primitives corresponding to an object in athree-dimensional (3D) space; and determining a respective EM fieldcontribution from the particular primitive to each of a plurality ofelements of a display by taking into consideration of an effect of theplurality of DCT weights of the image.

In some implementations, the method further includes: determining aresolution for the image to be mapped on the specified surface of theparticular primitive; and determining the plurality of DCT weights ofthe image based on the resolution.

In some implementations, the method further includes decoding the DCTweights of the image to obtain a respective DCT amplitude for each pixelof the image.

In some implementations, the method further includes storing valuesassociated with the respective DCT amplitudes of the pixels of the imagetogether with primitive data of the particular primitive. Determiningthe respective EM field contribution can include calculating therespective EM field contribution from the particular primitive to eachof the plurality of elements with the values associated with therespective DCT amplitudes of the pixels of the image.

In some implementations, the method further includes selectingparticular DCT terms to be included in the determining of the respectiveEM field contribution, each of the particular DCT terms having arespective DCT weight higher than a predetermined threshold.

Another aspect of the present disclosure features a method thatincludes: obtaining information of a given primitive and an occluder ofthe given primitive, wherein the given primitive is within a pluralityof primitives corresponding to an object in a three-dimensional (3D)space; and determining one or more particular elements of a plurality ofelements of a display that do not contribute to a reconstruction of thegiven primitive at an effect of the occluder.

In some implementations, the method further includes storing theinformation of the particular elements with the information of the givenprimitive and the occluder.

In some implementations, the determining is performed during a renderingprocess of the object for obtaining primitive data of the plurality ofprimitives.

In some implementations, the method further includes transmitting thestored information of the particular elements with the information ofthe given primitive and the occluder to a controller configured tocalculate electromagnetic (EM) contributions for the plurality ofprimitives to the plurality of elements of the display.

In some implementations, the method further includes, for each one ofthe particular elements, generating a sum of electromagnetic (EM) fieldcontributions from the plurality of primitives to the one of theparticular elements by excluding an EM field contribution from the givenprimitive to the one of the particular elements.

In some implementations, the method further includes, for each of theplurality of elements other than the particular elements, generating arespective sum of EM field contributions from the plurality ofprimitives to the element.

In some implementations, the method further includes masking an EM fieldcontribution of the particular elements to the given primitive.

In some implementations, determining the one or more particular elementsincludes: connecting the given primitive to endpoints of the occluder;extending the connection to the display to determine intersectionsbetween the connection and the display; and determining a particularrange defined by the intersections to be the particular elements that donot contribute to the reconstruction of the given primitive at theeffect of the occluder.

Another aspect of the present invention features a method that includes:obtaining information of a given primitive and an occluder of the givenprimitive, wherein the given primitive is within a plurality ofprimitives corresponding to an object in a three-dimensional (3D) space;and for each of a plurality of elements of a display, determining arespective part of the given primitive that does not make anelectromagnetic (EM) field contribution to the element at an effect ofthe occluder.

In some implementations, the method further includes storing theinformation of the respective part of the given primitive with theinformation of the given primitive and the occluder.

In some implementations, the determining is performed during a renderingprocess of the object for obtaining primitive data of the plurality ofprimitives.

In some implementations, the method further includes transmitting thestored information of the respective part of the given information withthe information of the given primitive and the occluder to a controllerconfigured to calculate electromagnetic (EM) contributions for theplurality of primitives to the plurality of elements of the display.

In some implementations, the method further includes masking an EM fieldcontribution of each of the plurality of elements to the respective partof the given primitive.

In some implementations, the method further includes, for each of theplurality of elements, generating a sum of EM field contributions fromthe plurality of primitives to the element by excluding an EM fieldcontribution from the respective part of the given primitive to theelement. Generating the sum of EM field contributions from the pluralityof primitives to the element can include subtracting the EM contributionof the respective part of the given primitive to the element from thesum of EM field contributions from the plurality of primitive to theelement without the effect of the occluder. Generating the sum of EMfield contributions from the plurality of primitives to the element caninclude summing EM field contributions from one or more other parts ofthe given primitive to the element, the respective part and the one ormore other parts forming the given primitive.

In some implementations, determining a respective part of the givenprimitive that do not make an EM field contribution to the element at aneffect of the occluder includes: connecting the element to endpoints ofthe occluder; determining intersections between the connection and thegiven primitive; and determining a particular part of the givenprimitive that is enclosed by the intersections to be the respectivepart of the given primitive that does not make the EM field contributionto the element at the effect of the occluder.

Another aspect of the present disclosure features a method that includesobtaining respective primitive data of each of a plurality of primitivescorresponding to an object in a three-dimensional (3D) space; obtainingrespective geometric specular information for each of the plurality ofprimitives; and storing the respective geometric specular informationwith respective primitive data for each of the plurality of primitives.

In some implementations, the respective geometric specular informationfor each of the plurality of primitives includes a reflectivity of asurface of the primitive upon a viewing angle.

In some implementations, the method further includes determining arespective EM field contribution from each of the plurality ofprimitives to each of a plurality of elements of a display by takinginto consideration of the respective geometric specular information forthe primitive.

Another aspect of the present disclosure features a method thatincludes: obtaining graphic data comprising respective primitive datafor a plurality of primitives corresponding to an object in athree-dimensional (3D) space; determining, for each of the plurality ofprimitives, an electromagnetic (EM) field contribution to each of aplurality of elements of a display by calculating, in a 3D coordinatesystem, an EM field propagation from the primitive to the element;generating, for each of the plurality of elements, a sum of the EM fieldcontributions from the plurality of primitives to the element;transmitting, for each of the plurality of elements, a respectivecontrol signal to the element, the control signal being for modulatingat least one property of the element based on the sum of the EM fieldcontributions to the element; and transmitting a timing control signalto an illuminator to activate the illuminator to illuminate light on thedisplay such that the light is caused by the modulated elements of thedisplay to form a volumetric light field corresponding to the object.

Another aspect of the disclosure features a method that includes: foreach of a plurality of elements of a display, altering a respectivecontrol signal with a predetermined calibration value; applying therespective altered respective control signals to the plurality ofelements of the display; measuring an output of light incident on thedisplay; and evaluating the predetermined calibration value based on themeasurement of the output of the light.

In some implementations, the predetermined calibration value is the samefor each of the plurality of elements.

In some implementations, the method further includes converting therespective control signals of the plurality of elements by adigital-to-analog converter (DAC), wherein altering the respectivecontrol signals for the plurality of elements includes altering digitalsignals of the respective control signals with the predeterminedcalibration value.

In some implementations, the predetermined value comprises a pluralityof bits.

In some implementations, the method further includes adjusting thepredetermined calibration value based on a result of the evaluation.Adjusting the predetermined calibration value can include modifying oneor more values of the plurality of bits. Adjusting the predeterminedcalibration value can include determining a combination of values of theplurality of bits based on the predetermined calibration value andanother calibration value determined from a previous evaluation.

In some implementations, the output of the light comprises a phasechange of the light or an intensity difference between the output of thelight and a background.

In some implementations, the respective control signal of the element isdetermined based on a sum of electromagnetic (EM) field contributionsfrom a plurality of primitives corresponding to an object to the elementin a 3D space.

Another aspect of the disclosure features a method that includes, foreach of a plurality of elements of a display: obtaining a respective sumof electromagnetic (EM) field contributions from a plurality ofprimitives in a three-dimensional (3D) space, the plurality ofprimitives corresponding to an object in the 3D space; applying arespective mathematical transform to the respective sum of EM fieldcontributions for the element to obtain a respective transformed sum ofEM field contributions for the element; determining a respective controlsignal based on the respective transformed sum of EM filed contributionsfor the element; and modulating a property of the element based on thedetermined respective control signal for the element.

In some implementations, the method further includes: introducing lightincident on the plurality of elements of the display; measuring a firstoutput of the light; and adjusting one or more coefficients of therespective mathematical transforms of the plurality of elements based ona result of the measurement of the first output of the light. The methodcan further include: changing a depth of a holographic patterncorresponding to the object in view of the display; measuring a secondoutput of the light; and adjusting the one or more coefficients of therespective mathematical transforms based on the first and secondoutputs. The method can further include: changing the plurality ofprimitives corresponding to a first holographic pattern to a secondplurality of primitives corresponding to a second holographic pattern;measuring a second output of the light; and adjusting the one or morecoefficients of the respective mathematical transforms based on thefirst and second outputs. The first holographic pattern and the secondholographic pattern can correspond to the object. The second holographicpattern can correspond to a second object different from the objectrelated to the first holographic pattern. The first output of the lightcan be measured by an imaging sensor. The imaging sensor can beconfigured to use a machine vision algorithm to determine what is beingdisplayed and calculate a fitness parameter. Each of the first andsecond holographic patterns can include a grid of dots, wherein thefitness parameter is at least one of: how close the dots are together;how centered the dots are positioned, and how much distortion the dotsare.

In some implementations, the mathematical transform is derived from aZernike polynomial expression.

In some implementations, the mathematical transforms for the pluralityof elements vary element-by-element.

In some implementations the method further includes: reproducing asample set of known colors and intensities by illuminating the display;measuring an output light using a colorimeter device calibrated to CIEstandard observer curves; and defining the output light of the displayin CIE XYZ color space. The method can further include: determining adeviation of values of the defined output light from known standardvalues; and adapting output colors on the display to bring them backinto alignment.

Another aspect of the disclosure features a method that includes:determining a cell gap of a liquid crystal (LC) display based on a pitchof display elements of the LC display; and calculating a minimum valueof a birefringence of LC mixture based on the cell gap and apredetermined retardance for the LC display.

In some implementations, the method further includes improving aswitching speed of the LC display with keeping the birefringence of LCmixture above the minimum value. Improving the switching speed caninclude at least one of: increasing dielectric anisotropy of the LCmixture; and decreasing the rotational viscosity of the LC mixture.

In some implementations, the LC display includes a liquid crystal onsilicon (LCOS) device having a silicon backplane.

In some implementations, the LC display includes: a liquid crystallayer; a transparent conductive layer on top of the liquid crystal layeras a common electrode; and a backplane comprising a plurality of metalelectrodes on bottom of the liquid crystal layer, wherein each of theplurality of metal electrodes is isolated from each other, and thebackplane is configured to control a voltage of each of the plurality ofmetal electrode.

Another aspect of the disclosure features a display that includes: abackplane; and a plurality of display elements on the backplane, whereinat least two of the plurality of display elements have different sizes.

In some implementations, a larger one of the at least two displayelements comprises a buffer, and a smaller one of the at least twodisplay elements comprises no buffer. The larger display element can beconnected with a first plurality of display elements by a conductiveline, wherein the buffer is configured to buffer a voltage applied onthe conductive line such that the voltage is only applied to a secondplurality of display elements within the first plurality of displayelements, a number of the second plurality of display elements beingsmaller a number of the first plurality of display elements.

In some implementations, the buffer comprises an analog circuit in aform of a transistor or a digital circuit in a form of logic gates.

In some implementations, a size distribution of the plurality of displayelements is substantially identical to a size of a smaller one of the atleast two display elements.

In some implementations, the display is configured to be a liquidcrystal on silicon device.

Another aspect of the disclosure features a display that includes: abackplane; and a plurality of display elements on the backplane, whereinat least two of the plurality of display elements have different shapes.

In some implementations, the backplane includes a respective circuit foreach of the display elements, wherein the respective circuits for the atleast two display elements have shapes corresponding to the differentshapes of the at least two display elements.

In some implementations, a size distribution of the plurality of displayelements is substantially identical to a predetermined size.

In some implementations, the display is configured to be a liquidcrystal on silicon device.

Another aspect of the present disclosure features a method including:obtaining graphic data including respective primitive data for aplurality of primitives corresponding to an object in athree-dimensional (3D) space; determining, for each of the plurality ofprimitives, an electromagnetic (EM) field contribution to each of aplurality of elements of a display by calculating, in a 3D coordinatesystem, an EM field propagation from the primitive to the element;generating, for each of the plurality of elements, a sum of the EM fieldcontributions from the plurality of primitives to the element;transmitting, for each of the plurality of elements, a respectivecontrol signal to the element, the control signal being for modulatingat least one property of the element based on the sum of the EM fieldcontributions to the element; and transmitting a timing control signalto an illuminator to activate the illuminator to illuminate light on thedisplay such that the light is caused by the modulated elements of thedisplay to form a volumetric light field corresponding to the object.

Other embodiments of the aspects include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.For a system of one or more computers to be configured to performparticular operations or actions means that the system has installed onit software, firmware, hardware, or a combination of them that inoperation cause the system to perform the operations or actions. For oneor more computer programs to be configured to perform particularoperations or actions means that the one or more programs includeinstructions that, when executed by data processing apparatus, cause theapparatus to perform the operations or actions.

Another aspect of the present disclosure features a device thatincludes: one or more processors; and a non-transitory computer readablestorage medium in communication with the one or more processors andstoring instructions executable by the one or more processors and uponsuch execution cause the one or more processors to perform one or moreof the methods disclosed herein.

Another aspect of the present disclosure features a non-transitorycomputer readable storage medium storing instructions executable by oneor more processors and upon such execution cause the one or moreprocessors to perform the method according to one or more of the methodsdisclosed herein.

Another aspect of the present disclosure features a display including aplurality of elements; and a controller coupled to the display andconfigured to perform one or more of the methods disclosed herein. Thecontroller can include a plurality of computing units, each of thecomputing units being configured to perform operations on one or moreprimitives of a plurality of primitives correspond to an object in athree-dimensional (3D) space. In some implementations, the controller islocally coupled to the display, and each of the computing units iscoupled to one or more respective elements of the display and configuredto transmit a respective control signal to each of the one or morerespective elements. The computing units can be configured to operate inparallel.

The controller can include at least one of an application-specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), aprogrammable gate array (PGA), a central processing unit (CPU), agraphics processing unit (GPU), or standard computing cells. The displaycan include a spatial light modulator (SLM) including a digitalmicro-mirror device (DMD) or a liquid crystal on silicon (LCOS) device.The display can be configured to be phase modulated, amplitudemodulated, or phase and amplitude modulated. The controller can becoupled to the display through a memory buffer.

In some implementations, the system includes an illuminator arrangedadjacent to the display and configured to emit light on the display. Theilluminator can be coupled to the controller and configured to be turnedon/off based on a control signal from the controller.

In some cases, the illuminator is coupled to the controller through amemory buffer configured to control amplitude or brightness of one ormore light emitting elements in the illuminator. The memory buffer forthe illuminator can have a smaller size than a memory buffer for thedisplay. A number of the light emitting elements in the illuminator canbe smaller than a number of the elements of the display. The controllercan be configured to simultaneously activate the one or more lightemitting elements of the illuminator.

The illuminator can be a coherent light source, a semi-coherent lightsource, or an incoherent light source. In some implementations, theilluminator is configured to emit a white light, and wherein the displayis configured to diffract the white light into light with differentcolors. In some implementations, the illuminator includes two or morelight emitting elements each configured to emit light with a differentcolor. The controller can be configured to sequentially modulate thedisplay with information associated with a first color during a firsttime period and modulate the display with information associated with asecond color during a second, sequential time period, and the controllercan be configured to control the illuminator to sequentially turn on afirst light emitting element to emit light with the first color duringthe first time period and a second light emitting element to emit lightwith the second color during the second time period.

In some implementations, the illuminator is arranged in front of asurface of the display and configured to emit the light on the surfaceof the display with an incident angle within a range between 0 degreeand 90 degree, and the emitted light is reflected from the surface ofthe display. In some cases, the emitted light from the illuminatorincludes collimated light. In some cases, the emitted light from theilluminator includes divergent light. In some cases, the emitted lightform the illuminator includes semi-collimated light.

In some implementations, the illuminator is arranged behind a rearsurface of the display and configured to emit a divergent light on therear surface of the display, and the emitted light is transmittedthrough the display and out of the display from a front surface of thedisplay.

In some implementations, the illuminator includes: a light sourceconfigured to emit the light; and a waveguide coupled to the lightsource and arranged adjacent to the display, the waveguide beingconfigured to receive the emitted light from the light source and guidethe emitted light to the display. In some cases, the light from thelight source is coupled to the waveguide from a side cross-section ofthe waveguide through a light coupler. In some cases, the light sourceand the waveguide are integrated in a planar form and positioned on asurface of the display. The waveguide can be configured to guide thelight to illuminate the display uniformly.

In some cases, the waveguide is positioned on a rear surface of thedisplay, and the light is guided to transmit through the display anddiffracted out of the display from a front surface of the display. Thecontroller can be positioned on a rear surface of the waveguide. In somecases, the waveguide is positioned on a front surface of the display,and wherein the light is guided to be incident on the front surface ofthe display and reflected by the front surface.

Another aspect of the present disclosure features a system including: adisplay including an array of elements; and an integrated circuitincluding an array of computing units, each of the computing units beingcoupled to one or more respective elements of the display and configuredto: compute an electromagnetic (EM) field contribution from at least oneprimitive of a plurality of primitives to each of the array of elements;and generate, for each of the one or more respective elements, arespective sum of the EM field contributions from the plurality ofprimitives to the element.

Each of the computing units can be configured to: receive, from othercomputing units of the array of computing units, computed EM fieldcontributions from other primitives of the plurality of primitives toeach of the one or more respective elements; and generate, for each ofthe one or more respective elements, the respective sum of the EM fieldcontributions by adding the received computed EM field contributionsfrom the other primitives to the element.

Each of the computing units can be configured to generate, for each ofthe one or more respective elements, a respective control signal tomodulate at least one property of the element based on the respectivesum of the EM field contributions to the element.

In some implementations, the integrated circuit includes a respectiveaccumulator configured to store an accumulation result of the computedEM field contribution from the plurality of primitives to each of theelements of the display. The integrated circuit can be configured toclean the accumulators at a beginning of a computation operation. Insome examples, the integrated circuit includes a respective memorybuffer for each of the elements, and the integrated circuit can beconfigured to accumulate the computed EM field contribution from theplurality of primitive to the element to obtain the respective sum ofthe EM field contributions as a final accumulation result in therespective accumulator and transfer the final accumulation result fromthe respective accumulator the respective memory buffer for the element.

In some implementations, the system further includes an illuminatorpositioned between the integrated circuit and the display and configuredto receive a control signal from the integrated circuit and illuminatelight on the display based on the control signal, and the integratedcircuit, the illuminator, and the display can be integrated as a singleunit.

A further aspect of the present disclosure features a system, including:a computing device configured to generate data including respectiveprimitive data of a plurality of primitives corresponding to an objectin a three-dimensional (3D) space; and the system as disclosed herein.The system is configured to receive the graphic data from the computingdevice and process the graphic data for presenting the object in the 3Dspace. The computing device can include an application programminginterface (API) configured to create the primitives with the respectiveprimitive data by rendering a computer generated (CG) model of theobject.

In the present disclosure herein, the term “primitive” refers to a basicnondivisible element for input or output within a computing system. Theelement can be a geometric element or a graphical element. The term“hologram” refers to a pattern displayed on a display which containsamplitude information or phase information, or some combination thereof,regarding an object. The term “holographic reconstruction” refers to avolumetric light field (e.g., a holographic light field) from a displaywhen illuminated.

The details of one or more implementations of the subject matter of thisspecification are set forth in the accompanying drawings and associateddescription. Other features, aspects, and advantages of the subjectmatter will become apparent from the description, the drawings, and theclaims.

It is to be understood that various aspects of implementations can becombined in different manners. As an example, features from certainmethods can be combined with features of other methods.

DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a schematic diagram of an example system including aholographic display.

FIG. 1B illustrates a schematic diagram of an example holographicdisplay.

FIG. 1C illustrates an example system for 3D displays.

FIG. 2 illustrates an example configuration for electromagnetic (EM)propagation calculation.

FIG. 3A illustrates an example EM propagation for a point primitiverelative to an element of a display.

FIG. 3B illustrates an example EM propagation for a line primitiverelative to an element of a display.

FIG. 3C illustrates an example EM propagation for a triangle primitiverelative to an element of a display.

FIG. 3D illustrates an example implementation of Maxwell holographicocclusion for a point primitive with a line primitive as an occluder.

FIG. 3E illustrates an example implementation of Maxwell holographicocclusion for a line primitive with another line primitive as anoccluder.

FIG. 3F illustrates an example implementation of Maxwell holographicocclusion for a triangle primitive with a line primitive as an occluder.

FIG. 3G illustrates an example implementation of Maxwell holographicstitching.

FIG. 4 is a flowchart of an example process of displaying an object in3D.

FIGS. 5A-5F show implementations of example systems for 3D displays.

FIG. 6A illustrates an example display with display elements havingnonuniform shapes.

FIG. 6B illustrates an example display with display elements havingdifferent sizes.

DETAILED DESCRIPTION

Implementations of the present disclosure feature technologies forenabling 3D displays of complex computer-generated scenes as genuineholograms. The technologies provide a novel and deterministic solutionto real time dynamic computational holography based upon Maxwell'sEquations for electromagnetic fields, which can be represented asMaxwell holography. The calculation (or computation) in the Maxwellholography can be represented as Maxwell holographic calculation (orMaxwell holographic computation). In embodiments, the disclosureapproaches a hologram as a Dirichlet or Cauchy boundary conditionproblem for a general electric field, utilizing tools including fieldtheory, topology, analytic continuation, and/or symmetry groups, whichenables to solve for holograms in real time without the limitations oflegacy holographic systems. In embodiments, the technologies can be usedto make phase-only, amplitude-only, or phase-and-amplitude holograms,utilizing spatial light modulators (SLMs) or any other holographicdevices.

Implementations of the present disclosure can provide: 1) a mechanism ofapproximation of a hologram as an electromagnetic boundary condition,using field theory and contact geometry, instead of classic optics; 2)derivation and implementation into computer codes and applicationprogramming interfaces (APIs) of the electromagnetic boundary conditionapproach to computational holography, that is, implementation of thehologram calculation as a 2D analytic function on a plane of thehologram and subsequent discretization into parallel algorithms; and/or3) implementation of a complete set of fully 3D, holographic versions ofstandard computer graphics primitives (e.g., point, line, triangle, andtexture triangle), which can enable full compatibility with standardexisting computer graphics tools and techniques. The technologies canenable devices to display general existing content that is notspecifically created for holography, and simultaneously allows existingcontent creators to create holographic works without having to learnspecial techniques, or use special tools.

Particularly, the technologies can involve the use of a mathematicalformulation (or expression) of light as an electromagnetic (EM)phenomenon in lieu of the mathematical formulation of classical opticsthat is commonly used in computational holography, e.g.,Gerchberg-Saxton (G-S) model. The mathematical formulation disclosedherein is derived from Maxwell's Equations. In embodiments, thetechnologies disclosed herein involve treating the displayed image as anelectromagnetic field and treating a hologram as a boundary valuecondition that produces the electromagnetic field (e.g., a Dirichletproblem). Additionally, a desired image can be constructed using aprimitive paradigm ubiquitous in computer graphics, allowing, forexample, the technologies to be used to display any 3D imagery as aholographic reconstruction, e.g., a holographic light field, instead ofas a projective image on a 2D screen. Compared to depth point cloudstechnologies that suffer from bandwidth limitations, the technologiescan avoid these limitations and use any suitable types of primitives,e.g., a point primitive, a line primitive, or a polygon primitive suchas a triangle primitive. Moreover, the primitives can be rendered withcolor information, texture information, and/or shading information. Thiscan help achieve a recording and compression scheme for CG holographiccontents including live holographic videos.

In embodiments, the technologies use Maxwell Equations to computegenerated holograms as a boundary condition problem for modeling of anelectromagnetic field, which can remove dependency on fast Fouriertransform (FFT) and its inherent limitations, remove dependency oncollimated light sources and lasers, and/or remove limitations ofprevious approaches to computational holography and non-deterministicsolutions.

In embodiments, the technologies can be optimized for computationalsimplicity and speed through a mathematical optimization process thatconstrains independent inputs to a surface of the hologram, depending onparameters of computer-generated (CG) primitives needed to build thescene. This allows work to be performed in a highly parallel and highlyoptimal fashion in computing architectures, e.g., application specificintegrated circuits (ASIC) and multicore architectures. The process ofcomputing the hologram can be considered as a single instruction thatexecutes on input data in a form of a computer-generated imagery (CGI)scene, and can theoretically be completed in a single clock cycle perCGI primitive.

In embodiments, the technologies treat a holographic scene as anassembly of fully 3D holographic primitive apertures which arefunctionally compatible with the standard primitives of conventional 3Dgraphics as employed in, for example, video games, movies, television,computer display, or any other computing display technologies. Thetechnologies can enable efficient implementation of these apertureprimitives in hardware and software without limitations inherent instandard implementations of computational holography. Amplitude andcolor of the primitives can be automatically computed. Computationalcomplexity can increase linearly with phase element number n, comparedto n{circumflex over ( )}2 or n*log(n) in standard computationalholography. The images created are fully 3D and not an assemblage ofplanar images, and the technologies do not require iterative amplitudecorrection with unknown step numbers. Moreover, the generated hologramsdo not have “conjugate” images that take up space on the holographicdevice.

As the holographic primitives are part of a special collection ofmathematical objects, they can be relatively simple and relatively fastto compute, and they can be uniquely suited to parallel, distributedcomputing approaches. The computability and parallelism can allow forinteractive computation of large holograms to design large areaholographic devices of theoretically unlimited size, which can act asholographic computer displays, phone displays, home theaters, and evenholographic rooms. Moreover, the holograms can fill large areas withlight, e.g., rendering large shaded areas in 3D, without limitationsassociated with conventional holographic computation methods which cancause elements to appear in outline instead of solid. Furthermore, therelatively simple and relatively fast computation allows for the displayof real-time holograms at interactive speeds that are not constrained byn{circumflex over ( )}2 computational load and by iterative amplitudecorrection.

In embodiments, the technologies can realize natural computability onmodern ASIC and multicore architectures and can realize completecompatibility with modern graphics hardware, modern graphics software,and/or modern graphics tools and tool chains. For example, thetechnologies can implement clear and simple holographic APIs and enablehigh performance rendering of arbitrary CG models using ordinary,standard 3D content creation tools, e.g., 3DS Max®, SolidWorks®, Maya®,or Unity3D, through the APIs. The APIs can enable developers or users tointeract with a holographic device, e.g., a light modulator orholographic system. The holographic APIs can create computer graphicsprimitives as discrete holographic scene primitives, allowing for richholographic content generation utilizing general purpose and speciallydesigned holographic computation hardware. The creation of amathematical and computational architecture can allow holograms to berendered using the tools and techniques used to make conventional 3Dcontent and software applications. The optimization of the mathematicaland computational architecture can allow for performant embodiments ofconventional graphics and rendering to be displayed as holographicreconstructions.

Algorithms in the technologies are relatively simple to implement inhardware. This not only allows the computational speeds needed for highquality, modern rendering that users expect, but it also allows thealgorithms to be implemented in relatively simple circuits, e.g., ASICgate structures, as part of a holographic device. Accordingly, bandwidthissues that can plague high density displays may become irrelevant, ascomputation of scenes can be spread across the computing architecturebuilt into the display device (e.g., built-in-computation) instead ofhaving to be computed remotely then written to each and every pixel ofthe display for each and every frame of content. It also means that thenumber of display elements, and thus the size of a holographic display,can be relatively unbound by constraints that limit other technologies.

The technologies can enable multiple interactive technologies usingstructured light to be implemented relatively simply and relativelyinexpensively in different applications, including, for example,solid-state light detection and ranging (LiDar) devices, 3D printing,smart illuminators, smart microdisplays, or any other applicationsdemanding structured light. The technologies can be also used foroptical simulations, e.g., for grating simulations.

FIG. 1A illustrates a schematic diagram of an example system 100 for 3Ddisplays. The system 100 includes a computing device 102 and aholographic display device (or a Maxwell holographic display device)110. The computing device 102 is configured to prepare data for a listof primitives corresponding to an object, e.g., a 3D object, andtransmit the data to the holographic display device 110 via a wired orwireless connection, e.g., USB-C connection or any other high speedserial connection. The holographic display device 110 is configured tocompute electromagnetic (EM) field contributions from the list ofprimitives to display elements of a display (e.g., a modulator) in theholographic display device 110, modulate the display elements with apattern, e.g., a hologram, based on the computed EM field contributionson the display, and display upon illumination a light fieldcorresponding to the object in 3D, e.g., a holographic reconstruction.Herein, the hologram refers to the pattern displayed on the displaywhich contains amplitude information or phase information, or somecombination thereof, regarding the object. The holographicreconstruction refers to a volumetric light field (e.g., a holographiclight field) from the display when illuminated.

The computing device 102 can be any appropriate type of device, e.g., adesktop computer, a personal computer, a notebook, a tablet computingdevice, a personal digital assistant (PDA), a network appliance, a smartmobile phone, a smartwatch, an enhanced general packet radio service(EGPRS) mobile phone, a media player, a navigation device, an emaildevice, a game console, or any appropriate combination of any two ormore of these computing devices or other computing devices.

The computing device 102 includes an operating system (OS) 104 that caninclude a number of applications 106 as graphics engines. Theapplications 106 can process or render a scene, e.g., any arbitrary CGmodel using standard 3D content creation tools, e.g., 3DS Max®,SolidWorks®, Maya®, or Unity3D. The scene can correspond to a 3D object.The applications 106 can operate in parallel to render the scene toobtain OS graphics abstraction 101 which can be provided to a graphicsprocessing unit (GPU) 108 for further processing. In someimplementations, the OS graphics abstraction 101 is provided to theholographic display device 110 for further processing.

The GPU 108 can include a specialized electronic circuit designed forrapid manipulation of computer graphics and image processing. The GPU108 can process the graphics abstraction 101 of the scene to getprocessed scene data 103 which can be used to obtain a list ofprimitives 105 indexed in a particular order. The primitives can includeat least one of a point primitive, a line primitive, or a polygonprimitive. In some implementations, the GPU 108 includes a video driverconfigured to generate the processed scene data 103 and the list ofprimitives 105.

In some implementations, the GPU 108 includes a conventional render 120,by which the list of primitives 105 can be rendered by conventionalrendering techniques, e.g., culling and clipping, into a list of itemsto draw on a conventional monitor 124, e.g., a 2D display screen. Thelist of items can be sent via a screen buffer 122 to the conventionalmonitor 124.

In some implementations, the GPU 108 includes a holographic renderer 130to render the list of primitives 105 into graphic data to be displayedby the holographic display device 110. The graphic data can include thelist of primitives and corresponding primitive data. For example, thegraphic data can include a hex code for each primitive.

In some implementations, the GPU 108 includes both the conventionalrender 120 and the holographic renderer 130. In some implementations,the GPU 108 includes the conventional renderer 120 and the holographicdisplay device 110 includes the holographic renderer 130.

The corresponding primitive data for a primitive can also include colorinformation (e.g., a textured color, a gradient color or both), textureinformation, and/or shading information. The shading information can beobtained by any customary CGI surface shading methods that involvemodulating color or brightness of a surface of the primitive.

The primitive data of a primitive can include coordinate information ofthe primitive in a 3D coordinate system, e.g., Cartesian coordinatesystem XYZ, polar coordinate system, cylindrical coordinate system, andspherical coordinate system. As discussed with further detail below, thedisplay elements in the holographic display device 110 can also havecorresponding coordinate information in the 3D coordinate system. Theprimitives at coordinate locations can represent a 3D object adjacent tothe display elements, e.g., in front of the display elements.

As an example, the primitive is a shaded line, e.g., a straight linethat changes smoothly from one color to another across its span. Theprimitive needs four elements of data to be rendered: two end points,and color information (e.g., a RGB color value) at each end point.Assume that a hex code for the line is AO, and the line stretches from afirst end point (0.1, 0.1, 0.1) to a second end point (0.2, 0.2, 0.2) inthe 3D coordinate system, with the color 1/2 Blue: RGB=(0,0,128) at thefirst end point and the color full Red: RGB=(255,0,0) at the second endpoint. The holographic renderer determines how much and what kind ofdata to expect for each primitive. For the line, the primitive data forthe shaded line in the primitive stream can be a set of instructions asbelow:

0xA0 // hex code for the shaded line 0x3dcccccd // first vertex at (0.1,0.1, 0.1) float (single) 0x3dcccccd 0x3dcccccd 0x000080 // first vertexcolor is (0, 0, 128) 0x3e4ccccd // second vertex at (0.2, 0.2, 0.2)float (single) 0x3e4ccccd 0x3e4ccccd 0xff0000 // second vertex color is(255, 0, 0)

There is a total of 31 hex words in the primitive data for the shadedline primitive.

Thus, it can be an extremely efficient way to transmit a complex scene,and the primitive data can further be compressed. Since each primitiveis a deterministic Turing step, there is no need for terminators.Different from a traditional model where this line primitive is simplydrawn on a 2D display screen, the primitive data for the line istransmitted to the holographic display device 110 that can compute ahologram and display a corresponding holographic reconstructionpresenting a line floating in space.

In some implementations, the computing device 102 transmitsnon-primitive based data, e.g., a recorded light field video, to theholographic display device 110. The holographic display device 110 cancompute sequential holograms to display the video as sequentialholographic reconstructions in space. In some implementations, thecomputing device 102 transmits CG holographic content simultaneouslywith live holographic content to the holographic display device 110. Theholographic display device 110 can also compute corresponding hologramsto display the contents as corresponding holographic reconstructions.

As illustrated in FIG. 1A, the holographic display device 110 includes acontroller 112 and a display 114. The controller 112 can include anumber of computing units or processing units. In some implementations,the controller 112 includes ASIC, field programmable gate array (FPGA)or GPU, or any combination thereof. In some implementations, thecontroller 112 includes the holographic renderer 130 to render the listof primitives 105 into the graphic data to be computed by the computingunits. In some implementations, the controller 112 receives the OSgraphics abstraction 101 from the computing device 102 for furtherprocessing. The display 114 can include a number of display elements. Insome implementations, the display 114 includes a spatial light modulator(SLM). The SLM can be a phase SLM, an amplitude SLM, or a phase andamplitude SLM. In some examples, the display 114 is a digitalmicro-mirror device (DMD) or a liquid crystal on silicon (LCOS) device.In some implementations, the holographic display device 110 includes anilluminator 116 adjacent to the display 114 and configured to emit lighttoward the display 114. The illuminator 116 can be a coherent lightsource, e.g., a laser, a semi-coherent light source, e.g., an LED (lightemitting diode), or an incoherent light source.

Different from a conventional 3D graphics system, which takes a 3D sceneand projects it onto a 2D display device, the holographic display device110 is configured to produce a 3D output such as a holographicreconstruction 117 in a form of a light field, e.g., a 3D volume ofcolor. In a hologram, each display element contributes to every part ofthe scene. That is, for the holographic display device 110, each displayelement needs to be modulated for every part of the scene, e.g., eachprimitive in the list of primitives generated by the GPU 108, forcomplete reproduction of the scene. In some implementations, modulationof certain elements may be omitted based on, for example, an acceptablelevel of accuracy in the reproduced scene.

In some implementations, the controller 112 is configured to compute anEM field contribution, e.g., phase, amplitude, or both, from eachprimitive to each display element, and generate, for each displayelement, a sum of the EM field contributions from the list of primitivesto the display element. This can be done either by running through everyprimitive and accruing its contribution to a given display element, orby running through each display element for each primitive.

The controller 112 can compute the EM field contribution from eachprimitive to each display element based on a predetermined expressionfor the primitive. Different primitives can have correspondingexpressions. In some cases, the predetermined expression is an analyticexpression, as discussed with further detail below in relation to FIGS.3A-3C. In some cases, the predetermined expression is determined bysolving Maxwell Equations with a boundary condition defined at thedisplay 114. The boundary condition can include a Dirichlet boundarycondition or a Cauchy boundary condition. Then, the display element canbe modulated based on the sum of the EM field contributions, e.g., bymodulating at least one of a refractive index, an amplitude index, abirefringence, or a retardance of the display element.

If values of an EM field, e.g., a solution to the Maxwell Equations, ateach point on a surface that bounds the field are known, an exact,unique configuration of the EM field inside a volume bounded by aboundary surface can be determined. The list of primitives (or aholographic reconstruction of a corresponding hologram) and the display114 define a 3D space, and a surface of the display 114 forms a portionof a boundary surface of the 3D space. By setting EM field states (e.g.,phase or phase and amplitude states) on the surface of the display 114,for example, by illuminating light on the display surface, the boundarycondition of the EM field can be determined. Due to time symmetry of theMaxwell Equations, as the display elements are modulated based on the EMfield contributions from the primitives corresponding to the hologram, avolumetric light field corresponding to the hologram can be obtained asthe holographic reconstruction.

For example, a line primitive of illumination at a specific color can beset in front of the display 114. As discussed in further detail belowwith respect to in FIG. 3B, an analytic expression for a linear aperturecan be written down as a function in space. Then the EM fieldcontribution from the line primitive on a boundary surface including thedisplay 114 can be determined. If EM field values corresponding to thecomputed EM field contribution are set on the display 114, due totime-symmetry of the Maxwell Equations, the same linear aperture used inthe computation can appear at a corresponding location, e.g., acoordinate position of the linear primitive in the 3D coordinate system.

In some examples, as discussed in further detail below with respect toFIG. 3B, suppose that there is a line of light between two points A andB in the 3D space. The light is evenly lit and has an intensity I perline distance l. At each infinitesimal dl along the line from A to B, anamount of light proportional to I*dl is emitted. The infinitesimal dlcan acts as a delta (point) source, and the EM field contribution fromthe infinitesimal dl to any point on a boundary surface around a scenecorresponding to a list of primitives can be determined. Thus, for anydisplay element on the display 114, an analytic equation that representsthe EM field contribution at the display element from the infinitesimalsegment of the line can be determined. A special kind ofsummation/integral that marches along the line and accrues the EM fieldcontribution of the entire line to the EM field at the display elementon the display can be determined as an expression. Values correspondingto the expression can be set at the display element, e.g., by modulatingthe display element and illuminating the display element. Then, throughtime reversal and a correction constant, the line can be created in thesame location defined by points A and B in the 3D space.

In some implementations, the controller 112 is coupled to the display114 through a memory buffer. The control signal 112 can generate arespective control signal based on the sum of the EM field contributionsto each of the display elements. The control signal is for modulatingthe display element based on the sum of the EM field contributions. Therespective control signals are transmitted to the corresponding displayelements via the memory buffer.

In some implementations, the controller 112 is integrated with thedisplay 114 and locally coupled to the display 114. As discussed withfurther detail in relation to FIG. 1B, the controller 112 can include anumber of computing units each coupled to one or more respective displayelements and configured to transmit a respective control signal to eachof the one or more respective display elements. Each computing unit canbe configured to perform computations on one or more primitives of thelist of primitives. The computing units can operate in parallel.

In some implementations, the illuminator 116 is coupled to thecontroller 112 and configured to be turned on/off based on a controlsignal from the controller 112. For example, the controller 112 canactivate the illuminator 116 to turn on in response to the controller112 completing the computation, e.g., all the sums of the EM fieldcontributions for the display elements are obtained. As noted above,when the illuminator 116 emits light on the display 114, the modulatedelements of the display cause the light to propagate in differentdirections to form a volumetric light field corresponding to the list ofprimitives that correspond to the 3D object. The resulting volumetriclight field corresponds to a solution of Maxwell's equations with aboundary condition defined by the modulated elements of the display 114.

In some implementations, the controller 112 is coupled to theilluminator 116 through a memory buffer. The memory buffer can beconfigured to control amplitude or brightness of light emitting elementsin the illuminator. The memory buffer for the illuminator 116 can have asmaller size than a memory buffer for the display 114. A number of thelight emitting elements in the illuminator 116 can be smaller than anumber of the display elements of the display 114, as long as light fromthe light emitting elements can illuminate over a total surface of thedisplay 114. For example, an illuminator having 64×64 OLEDs (organiclight emitting diodes) can be used for a display having 1024×1024elements. The controller 112 can be configured to simultaneouslyactivate a number of lighting elements of the illuminator 116.

In some implementations, the illuminator 116 is a monochromatic lightsource configured to emit a monochromatic light, e.g., a red light, agreen light, or a blue light. In some implementations, the illuminator116 includes two or more light emitting elements each configured to emitlight with a different color. For example, the illuminator 116 caninclude red, green, and blue lighting elements. To display a full-color3D object, three separate holograms for red, green, and blue, can becomputed. That is, three EM field contributions from correspondingprimitives to the display elements can be obtained. The display elementscan be modulated sequentially based on the three EM field contributionsand the illuminator 116 can be controlled to sequentially turn on thered, green and blue lighting elements. Depending on temporalcoherence-of vision effect in an eye of a viewer, the three colors canbe combined in the eye to give an appearance of full color. In somecases, the illuminator 116 is switched off during a state change of thedisplay image (or holograhic reconstruction) and switched on when avalid image (or holographic reconstruction) is presented for a period oftime. This can also depend on the temperal coherence of vision to makethe image (or holographic reconstruction) appear stable.

In some implementations, the display 114 has a resolution small enoughto diffract visible light, e.g., on an order of 0.5 μm or less. Theilluminator 116 can include a single, white light source and the emittedwhite light can be diffracted by the display 114 into different colorsfor holographic reconstructions.

As discussed in further detail below with respect to FIGS. 5A-5F, therecan be different configurations for the system 100. The display 114 canbe reflective or transmissive. The display 114 can have various sizes,ranging from a small scale (e.g., 1-10 cm on a side) to a large scale(e.g., 100-1000 cm on a side). Illumination from the illuminator 116 canbe from the front of the display 114 (e.g., for a reflective display) orfrom the rear of the display 114 (e.g., for a transmissive display). Aplanar waveguide can be used to evenly illuminate a surface of thedisplay 114. In some implementations, the controller 112, theilluminator 116, and the display 114 can be integrated together as asingle unit. The integrated single unit can include the holographicrenderer 130, e.g., in the controller 112.

FIG. 1B illustrates a schematic diagram of an example holographicdisplay device 150. The holographic display device 150 can be similar tothe holographic display device 110 of FIG. 1A. The holographic displaydevice 150 includes a computing architecture 152 and a display 156. Thecomputing architecture 152 can be similar to the controller 112 of FIG.1A. The computing architecture 152 can include an array of parallelcomputing cores 154. A computing core can be connected to an adjacentcomputing core via a communication connection 159, e.g., a USB-Cconnection or any other high speed serial connection. The connections159 can be included in a data distribution network through which scenedata 151 (e.g., scene primitives) can be distributed among the computingcores 154.

The display 156 can be similar to the display 114 of FIG. 1A, and caninclude an array of display elements 160 positioned on a backplane 158.The display elements 160 can be arranged on a front side of thebackplane 158 and the computing cores 154 can be arranged on a back sideof the backplane 158. The backplane 158 can be a substrate, e.g., awafer. The computing cores 154 can be either on the same substrate asthe display 156 or bonded to the back side of the display 156.

Each computing core 154 can be connected to a respective tile (or array)of display elements 160. Each computing core 154 is configured toperform computations on respective primitives of a number of primitivesin the scene data 151 in parallel with each other. In some examples, thecomputing core 154 is configured to compute an EM field contributionfrom each of the respective primitives to each of the array of displayelements 160 and generate a sum of EM field contributions from thenumber of primitives to each of the respective tile of display elements160. The computing core 154 can receive, from other computing cores ofthe array of computing cores 154, computed EM field contributions fromother primitives of the number of primitives to each of the respectivetile of display elements 160, and generate the sum of EM fieldcontributions based on the received computed EM field contributions. Thecomputing core 154 can generate a control signal for each of therespective tile of display elements to modulate at least one property ofeach of the respective tile of display elements 160 based on the sum ofEM field contributions to the display element.

As noted above, the computing architecture 152 can also generate acontrol signal to an illuminator 162, e.g., in response to determiningthat the computations of the sums of the EM field contributions from thenumber of primitives to each of the display elements have beencompleted. The illuminator 162 emits an input light 153 to illuminate onthe modulated display elements 160 and the input light 153 is reflectedby the modulated display elements 160 to form a volumetric light fielde.g., a holographic light field 155, corresponding to the scene data151.

As illustrated in FIG. 1B, the tiles of display elements 160 can beinterconnected into a larger display. Correspondingly, computing cores154 can be interconnected for data communication and distribution. Notethat a parameter that changes in the holographic calculations betweenany given two display elements is their physical locations. Thus, thetask of computing the hologram can be shared between the correspondingcomputing cores 154 equally, and the entire display 150 can operate atthe same speed as a single tile, independent of the number of tiles.

FIG. 1C illustrates an exemplary system 170 for displaying objects in a3D space. The system 170 can include a computing device, e.g., thecomputing device 102 of FIG. 1A, and a holographic display device 172,e.g., the holographic display 110 of FIG. 1A or 150 of FIG. 1B. A usercan use an input device, e.g., a keyboard 174 and/or a mouse 176, tooperate the system 170. For example, the user can create a CG model fora 2D object 178 and a 3D object 180 through the computing device. Thecomputing device or the holographic display device 172 can include aholographic renderer, e.g., the holographic renderer 130 of FIG. 1A, torender the CG model to generate corresponding graphic data for the 2Dobject 178 and the 3D object 180. The graphic data can includerespective primitive data for a list of primitives corresponding to theobjects 178 and 180.

The holographic display device 172 can include a controller, e.g., thecontroller 112 of FIG. 1A or 152 of FIG. 1B, and a display 173, e.g.,the display 114 of FIG. 1A or 156 of FIG. 1B. The controller can computea respective sum of EM field contributions from the primitives to eachdisplay element of the display 173 and generate control signals formodulating each display element based on the respective sum of EM fieldcontributions. The holographic display device 172 can further include anilluminator, e.g., the illuminator 116 of FIG. 1A or the illuminator 162of FIG. 1B. The controller can generate a timing control signal toactivate the illuminator. When light from the illuminator illuminates ona surface of the display 173, the modulated display elements can causethe light to propagate in the 3D space to form a volumetric light fieldcorresponding to a holographic reconstruction for the 2D object 178 anda holographic reconstruction for the 3D object 180. Thus, the 2D object178 and the 3D object 180 are displayed as respective holographicreconstructions floating in the 3D space in front of the display 173.

In some implementations, the computing device transmits non-primitivebased data, e.g., a recorded light field video, to the holographicdisplay device 172. The holographic display device 172 can compute andgenerate corresponding holograms, e.g., a series of sequentialholograms, to display as corresponding holographic reconstructions inthe 3D space. In some implementations, the computing device transmits aCG holographic content simultaneously with live holographic content tothe holographic display device 172. The holographic display device 172can also compute and generate corresponding holograms to display thecontents as corresponding holographic reconstructions in the 3D space.

FIG. 2 illustrates an exemplary configuration 200 for electromagnetic(EM) field calculation. A display 202, e.g., an LCOS device, includingan array of elements 204 and a list of primitives including a pointprimitive 206 are in a 3D space 208. The 3D space 208 includes boundarysurfaces 210. In a 3D coordinate system XYZ, the point primitive 206 hascoordinate information (x, y, z). Each display element 204 lies in aflat plane with respect to other display elements 204 and has a 2Dposition (u, v). The display element 204 also has a location in the 3Dspace. By a mathematical point transformation, the 2D position (u, v)can be transferred into six coordinates 250 in the 3D coordinate system.That is, a surface of the display 202 forms a portion of the boundarysurfaces 210. Thus, EM field contributions from the list of primitivesto a display element computed by defining a boundary condition at thesurface of the display 202 represent a portion of the total EM fieldcontributions from the primitives to the display element. A scalefactor, e.g., six, can be multiplied to a sum of the EM fieldcontributions for each of the display elements to obtain a scaled sum ofthe field contributions, and the display element can be modulated basedon the scaled sum of the field contributions.

Exemplary EM Field Contributions for Primitives

Primitives can be used for standard computer graphics rendering. Eachtype of primitive in standard computer graphics corresponds in thisformulation to a discrete mathematical function that defines a singleholographic primitive for a graphical element added to a hologram. Eachtype of primitive can correspond to an expression for calculating an EMfield contribution to a display element. A primitive can be a pointprimitive, a line primitive, or a polygon (e.g., a triangle) primitive.As illustrated below, an analytic expression can be derived bycalculating EM field propagation from a corresponding primitive to adisplay element of a display.

FIG. 3A illustrates an example EM propagation from a point primitive 304to an element 302 of a display 300. In a 3D coordinate system XYZ, it isassumed that z coordinate is 0 across the display 300, which meansnegative z values are behind the display 300 and positive z values arein front of the display 300. The point primitive 304 has a coordinate(x, y, z), and the display element 302 has a coordinate (u, v, 0). Adistance d_(uv) between the point primitive 304 and the display element302 can be determined based on their coordinates.

The point primitive 304 can be considered as a point charge with timevarying amplitude. According to electromagnetic theory, an electricfield E generated by such a point charge can be expressed as:

$\begin{matrix}{{{E} \propto \frac{\exp \left( {i\; 2\; \pi \; d\text{/}\lambda} \right)}{d^{2}}},} & (1)\end{matrix}$

where λ, represents a wavelength of an EM wave, and d represents adistance from the point charge.

Thus, the electric field E_(u,v) at the display element (u,v) can beexpressed as:

$\begin{matrix}{{{E_{u,v}} \propto {\frac{I}{d_{uv}^{2}}{\exp \left( {i\; 2\; \pi \; d_{uv}\text{/}\lambda} \right)}}},} & (2)\end{matrix}$

where I represents a relative intensity of the holographic primitiveelectric field at the display element contributed from the pointprimitive 304.

As discussed above with respect to FIG. 2, a surface of the display 300forms only a portion of a boundary surface for the EM field. A scalefactor

can be applied to the electric field E_(u,v) to get a scaled electricfield E_(φ)(u, v) at the display element that adjusts for the partialboundary as follows:

$\begin{matrix}{{{E_{\phi}\left( {u,v} \right)} \propto {\frac{\delta \; I}{d_{uv}^{2}}{\exp \left( {i\; 2\; \pi \; d_{uv}\text{/}\lambda} \right)}}},} & (3)\end{matrix}$

where δ=[6+ε],0<ε≤1.

FIG. 3B illustrates an example of EM propagation from a line primitive306 to the display element 302 of the display 300 in the 3D coordinatesystem XYZ. As noted above, the display element 302 can have acoordinate (u, v, 0), where z=0. The line primitive 306 has twoendpoints P₀ with coordinate (x₀, y₀, z₀) and P₁ with coordinate (x₁,y₁, z₁). A distance do between the endpoint P₀ and the display elementcan be determined based on their coordinates. Similarly, a distance d₁between the endpoint P₁ and the display element can be determined basedon their coordinates. A distance d₀₁ between the two endpoints P₀ and P₁can be also determined, e.g., d₀₁=d₁−d₀.

As discussed above, a line primitive can be treated as a superpositionor a linear deformation, and a corresponding analytic expression for theline primitive as a linear aperture can be obtained as a distributeddelta function in space. This analytic expression can be a closedexpression for continuous 3D line segments as holograms.

FIG. 3C illustrates an example EM propagation from a triangle primitive308 to the display element 302 of the display 300 in the 3D coordinatesystem XYZ. As noted above, the display element 302 can have acoordinate (u, v, 0), where z=0. The triangle primitive 308 has threeendpoints: P₀ (x₀, y₀, z₀), P₁ (x₁, y₁, z₁), and P₂ (x₂, y₂, z₂).Distance d₀, d₁, and d₂ between the display element and the endpointsP₀, P₁, and P₂ can be respectively determined based on theircoordinates.

Similar to the line primitive in FIG. 3B, the triangle primitive can betreated as a continuous aperture in space and an analytical expressionfor the EM field contribution of the triangle primitive to the displayelement can be obtained by integration. This can be simplified to obtainan expression for efficient computation.

Exemplary Computations for Primitives

As discussed above, a controller, e.g., the controller 112 of FIG. 1A,can compute an EM field contribution from a primitive to a displayelement based on an analytical expression that can be determined asshown above. As an example, the EM field contribution for a lineprimitive is computed as below.

Each display element in a display has a physical location in space, andeach display element lies in a flat plane with respect to other displayelements. Assuming that the display elements and their controllers arelaid out as is as customary in display and memory devices, a simplemathematical point transformation can be used to transform a logicallocation of a given display element based on a logical memory addressfor the display element in a processor to an actual physical location ofthe display element in the space. Therefore, as the logical memoryaddresses of the display elements are looped in a logical memory spaceof the processor, corresponding actual physical locations in the spaceacross the surface of the display can be identified.

As an example, if the display has a 5 μm pitch, each logical addressincrement can move 5 μm in the x direction, and when x resolution limitof the display is reached, the next increment will move back to theinitial x physical location and increment the y physical location by 5μm. The third spatial coordinate z can be assumed to be zero across thedisplay surface, which means that the negative z values are behind thedisplay, and the positive z values are in front of the display.

To begin the line calculation, a type of scaled physical distancebetween the current display element and each of the two points of theline primitive can be determined to be d0 and d1. As a matter of fact,d0 and d1 can be calculated once per primitive, as every subsequentcalculation of the distances across display elements is a smallperturbation of an initial value. In this way, this computation isperformed in one dimension.

An example computation process for each primitive can include thefollowing computation codes:

DD=f(d1, d0),

iscale=SS*COLOR*Alpha1,

C1=−2*iscale*sin(DD/2)*sin((Alpha2)*cos(Alpha3),

C2=−2*iscale*sin(DD/2)*sin(Alpha2)*sin(Alpha4),

where SS, Alpha1, Alpha2, Alpha3, and Alpha4 are pre-computed constants,COLOR is the RGB color value passed in with the primitive, and allvalues are scalar, single precision floats. Both the sine and cosinefunctions can be looked up in tables stored in the controller to improvecomputation efficiency.

The results in C1 and C2 are then accumulated for each primitive at eachdisplay element, e.g., in an accumulator for the display element, andcan be normalized once at the end of the computations for the displayelements. At this point, as noted above, the controller can transmit afirst control signal to the display elements to modulate the displayelements based on the computed results and a second control signal to anilluminator to turn on to emit light. Accordingly, a holographicreconstruction (or a holographic light field) is visible to a viewer.When illuminated, the modulated display elements can cause the light toproduce a crisp, continuous color line in three dimensional space.

In some implementations, the computation codes include a hex code forcleaning previous accumulations in the accumulator, e.g., at thebeginning of the codes. The computation codes can also include a hexcode for storing the accumulator results into a respective memory bufferfor each display element, e.g., at the end of the codes. In someimplementations, a computing device, e.g., the computing device 102 ofFIG. 1A, transmits a number of background or static primitive hex codesto the controller at an application startup or an interval betweendisplaying frames that does not affect a primary display frame rate. Thecomputing device can then transmit one or more combinations of the hexcodes potentially along with other foreground or dynamic primitives at amuch higher rate to the controller that can form a corresponding controlsignal to modulate the display elements of the display.

The computation process can be orders of magnitude simpler and fasterthan the most efficient line drawing routines in conventional 2D displaytechnology. Moreover, this computation algorithm scales linearly withthe number of display elements. Thus, scaling computing units of thecontroller as a 2D networked processing system can keep up withcomputation needs of an increasing surface area of the display.

Exemplary Computation Implementations

A Maxwell holographic controller, e.g., the controller 112 of FIG. 1A,can compute an EM field contribution from a primitive to a displayelement based on an analytical expression that can be determined asshown above. The controller can be implemented in, for example, ASIC,FPGA or GPU, or any combination thereof.

In a modern GPU pipeline, a GPU takes descriptions of geometric figuresas well as vertex and fragment shader programs to produce color anddepth pixel outputs to one or more output image surfaces (called rendertargets). The process involves an explosive fan-out of information wheregeometry is expanded into shading fragments, followed by a visibilitytest to select whether work needs to be done on each of these fragments.A fragment is a record that contains all the information involved toshade that sample point, e.g., barycentric coordinates on the triangle,interpolated values like colors or texture coordinates, surfacederivatives, etc. The process of creating these records then rejectingthose that do not contribute to the final image is the visibility test.Fragments that pass the visibility test can be packed into work groupscalled wavefronts or warps that are executed in parallel by the shaderengines. These produce output values that are written back to memory aspixel values, ready for display, or for use as input textures for laterrendering passes.

In Maxwell holography, the rendering process can be greatly simplified.In Maxwell holographic calculations, every primitive contributes toevery display element. There is no need to expand geometry into pixelsand no need to apply visibility tests before packing wavefronts. Thiscan also remove the need for decision making or communication betweenMaxwell holographic pipelines and allow computation to become a parallelissue with a number of possible solutions each one tuned to speed, cost,size or energy optimization. The graphics pipeline is significantlyshorter with less intermediate steps, no data copying or movement, andless decisions leading to lower latency between initiating a draw andthe result being ready for display. This can allow Maxwell holographicrendering to create extremely low latency displays. As discussed below,this can allow Maxwell holographic calculations to increase accuracy,for example, by using fixed point numbers in the Maxwell holographicpipeline, and to optimize computation speed, for example, by optimizingmathematical functions.

Using Fixed Point Numbers

When calculating an EM contribution from each primitive at each displayelement (e.g., phasel), intermediate calculations involve producing verylarge numbers. These large numbers involve special handling as they alsoneed to retain the fractional parts during the calculation.

Floating point values have the disadvantage that they are most accurateclose to the origin (zero on the number line) and lose one bit ofaccuracy every power-of-two when moving away from the origin. Fornumbers close in the range [−1,1], the accuracy of floating point valuescan be exquisite, but once reaching numbers in the tens of millions,e.g., reaching the point where single-precision 32-bit IEEE-754 floatingpoint values have no fractional digits remaining, the entire significand (a.k.a mantissa) is used to represent the integer part of the value.However, it is the fractional part of large numbers that Maxwellholography is particularly interested in retaining.

In some cases, fixed point numbers are used in the Maxwell holographiccalculations. Fixed point number representations are numbers where thedecimal point does not change on a case-by-case basis. By choosing thecorrect numbers of bits for the integer and fractional parts of anumber, the same number of fractional bits can be obtained regardless ofthe magnitude of the number. Fixed point numbers are represented asintegers with an implicit scale factor, e.g., 14.375 can be representedas the number 3680 (0000111001100000 base-2) in a 16-bit fixed pointvalue with 8 fractional bits. This can be also represented as an“unsigned 16.8” fixed point number, or u16.8 for short. Negative numberscan have one additional sign bit and are stored in “2s compliment”format. In such a way, the accuracy of the calculation can be greatlyimproved.

Optimization to Mathematical Functions

As shown above, Maxwell holographic calculations involve the use oftranscendental mathematical functions, e.g., sine, cosine, arc tangent,etc. In a CPU, these functions are implemented as floating point libraryfunctions that may use specialized CPU instructions, or on a GPU asfloating point units in the GPU. These functions are written to takearguments as a floating point number and the results are returned in thesame floating point representation. These functions are built for thegeneral case, to be accurate where floats are accurate, to be correctlyrounded and to cope with every edge case in the floating point numberrepresentation (+/−Infinity, NaN, signed zero and denormal floats).

In Maxwell holographic calculations, with the fixed pointrepresentation, there is no need to use denormal floats for gradualunderflow, no handling NaN results from operations like division byzero, no altering the floating point rounding modes, and no need toraise floating point exceptions to the operating system. All of theseallow simplifying (and/or optimizing) the transcendental mathematicalfunctions, for example, as discussed below.

In some cases, optimizations can be made to take arguments in one fixedpoint format and return the value to a different level of accuracy,e.g., input s28.12 and output s15.14. This can be especially desirablewhen calculating the sine of large values in the 10s of millions, theinput argument may be large but the output can only need to representthe value range [−1,1], or arctangent which takes in any value butreturn values in the range [−π/2, π/2].

In some cases, optimization can be made to freely implement thetranscendental functions as fully enumerated look-up tables,interpolated tables, as semi-table based polynomial functions or assemi-table based full minimax polynomials depending on the input rangeinvolved. It also allows to apply specialized range reduction methodsthat cope with large inputs, which the general purpose GPU pipelinecalculation may skip for speed.

In some cases, another optimization can be transforming trigonometriccalculations from the range [−π, π] into a signed 2's complimentrepresentation in the range [−1,1] which has the advantage of notrequiring expensive modulo a division operations.

Exemplary Implementations for Occlusion

Occlusion is often viewed as a di□cult and important topic in computergraphics, and even more so in computational holography. This is because,in at least some cases, while the occlusion problem in projective CGI isstatic, what is hidden and what is visible in holographic systems dependon the location and direction of a viewer. Wave approaches of G-Sholography or its derivatives have been developed to address theholographic occlusions. However, masking or blocking contributions fromparts of a scene that are behind other parts of a scene can be verycomplicated and computationally expensive in the G-S methodology.

In Maxwell holography, the occlusion issue can be addressedcomparatively easily, because which display elements (e.g., phasels)correspond to which primitives is completely deterministic and trivial.For example, whether or not a given display element contributes to areconstruction of a given primitive can be determined as the calculationfor the given primitive is performed. After determining that a number ofdisplay elements do not contribute to the given primitive due toocclusion, when calculating a sum of EM contributions to one of thenumber of display elements, the EM contribution from the given primitiveis omitted from the calculation of the sum of EM contributions to theone of the number of display elements.

For illustration only, FIGS. 3D-3F show a determination of displayelements not contributing to a given primitive (a point in FIG. 3D, aline in FIG. 3E, and a triangle in FIG. 3F) with a line primitive as anoccluder. The line primitive has a starting point O1 and an ending pointO2.

As illustrated in FIG. 3D, a point primitive P0 is behind the occluderand closer to the display. By extending lines connecting O1-P0 andO2-P0, a range of display elements from D1 to D2 in the display isdetermined, which do not contribute to the reconstruction of the pointprimitive P0.

In some examples, the coordinate information of O1, O2, and P0 is known,e.g., stored in a “Z” buffer calculated by a GPU (e.g., the GPU 108 ofFIG. 1A) prior to the scene being transmitted to the Maxwell holographiccontroller (e.g., the controller 112 of FIG. 1A). For example, in an XZplane with y=0, the coordinate information can be O1 (Ox1, Oz1), O2(Ox2, Oz2), and P0 (Px, Pz), with Oz1=Oz2=Oz. Based on the coordinateinformation, the coordinate information of D1 and D2 can be determinedto be

Dx1=Px+ρ(Px−Ox2), Dx2=Dx1+ρ(Ox2−Ox1)  (4),

where ρ=Pz/(Oz−Pz), and Dz1=Dz2=0.

The information of D1 and D2 can be stored as additional information inan “S” buffer for the Maxwell holographic controller, besides theinformation in a Z buffer for the point primitive P0. In such a way, theadditional information can be used to trivially mask the contributionsof specific display elements (within the range from D1 to D2) to thespecific primitive P0 in the indexed primitive list.

FIG. 3E illustrates a determination of how a specific display elementcontributes to a line primitive with an occluder before the lineprimitive. By connecting the specific display element D0 to the startingpoint O1 and the ending point O2 of the occluder, two point primitivesP1 and P2 on the line primitive are determined as the intersectionpoints. Thus, the specific display element D0 does not contribute to thereconstruction of the part of the line primitive from P1 to P2 on theline primitive. Accordingly, when calculating the sum of EMcontributions to the specific display element D0, the EM contributionsfrom the part P1-P2 of the line primitive is not calculated.

This can be implemented in two ways. In the first way, the EMcontributions from the part P0-P1 and the part P2-Pn to the specificdisplay element D0 are summed as the EM contributions of the lineprimitive to the specific display element D0, by considering theocclusion from the occluder. In the second way, the EM contribution fromthe whole line primitive P0-Pn is calculated, together with the EMcontribution from the part P1-P2, a difference between the twocalculated EM contributions can be considered as the EM contribution ofthe line primitive to the specific display element D0 by considering theocclusion from the occluder. The coordinate information of P1 and P2 orthe part P1-P2 can be stored, as the part of the line primitive thatdoes not contribute to the specific display element D0, in the “S”buffer of the Maxwell holographic controller, together with theinformation of the occluder and other information in the “Z” buffer ofthe GPU.

FIG. 3F illustrates a determination of how a specific display elementcontributes to a triangle primitive with an occluder before the triangleprimitive. By connecting the specific display element D0 to the startingpoint O1 and the ending point O2 of the occluder, four point primitivesP1, P2, P3, and P4 on sides of the triangle primitive are determined asthe intersection points. Thus, the specific display element D0 does notcontribute to the reconstruction of the part of the triangle primitiveenclosed by the points P1, P2, P3, P4, P_(C). Accordingly, whencalculating the sum of EM contributions to the specific display elementD0, the EM contributions from the part P1-P2-P3-P4-P_(C) of the triangleprimitive is not calculated. That is, only the EM contributions from thefirst triangle formed by points P_(A), P1 and P2 and the second triangleformed by points P_(B), P3, and P4 are summed as the EM contribution ofthe triangle primitive P_(A)-P_(B)-P_(C) by considering the occlusion ofthe occluder. The coordinate information of P1, P2, P3 and P4 or thetriangle primitives P_(A)-P1-P2 and P_(B)-P3-P4 can be stored, as thepart of triangle primitive P_(A)-P_(B)-P_(C) that contributes to thespecific display element D0, in the “S” buffer of the Maxwellholographic controller, together with the information of the occluderand other information in the “Z” buffer of the GPU.

The implementations of occlusion in Maxwell holography enables toconvert the “Z” buffer in the GPU to the “S” buffer in the Maxwellholographic controller, and can mask the contributions of specificprimitives (or specific parts of the primitives) in the indexedprimitive list to a specific display element. This not only providesaccurate, physically correct occlusion, it also saves computation time,as the primitives that do not contribute to a given display element andmove on to computation for the next display element. The “S” buffer cancontain additional information related to diffraction efficiency of thedisplay.

The “S” buffer can also include rendering features such as Holographicspecular highlights, in which a reflectivity of a surface is dependentupon the viewing angle. In traditional CGI, specular highlights aredependent only on the orientation of the rendered object, whereas in aMaxwell holographic context, the direction from which the object isviewed also plays a part. Therefore, the geometric specular informationcan be encoded in the “S” buffer as an additive (specular) rather than asubtractive (occlusion) contribution. In Maxwell holography, themathematics for holographic specular highlights can be substantiallysame as that for holographic occlusion.

Exemplary Implementations for Stitching

When light illuminates on a display modulated with EM contributions froma list of primitives of a 3D object, the modulated display causes thelight to propagate in different directions to form a volumetric lightfield corresponding to the primitives. The volume light field is theMaxwell holographic reconstruction. Two adjacent primitives in the 3Dobject, e.g., triangle primitives, have a shared side. During thereconstruction, a stitching issue may raise, where the light intensityof the shared side can be double due to the reconstructions of the twoadjacent primitives separately. This may affect the appearance of thereconstructed 3D object.

To address the stitching issue in Maxwell holography, as illustrated inFIG. 3G, the adjacent primitives can be scaled down by a predeterminedfactor, so that a gap can be formed between the adjacent primitives. Insome cases, instead of scaling down the two adjacent primitives, onlyone primitive or a part of the primitive is scaled down. For example, aline of a triangle primitive can be scaled down to separate from anothertriangle primitive. In some cases, the scaling can include scalingdifferent parts of a primitive with different predetermined factors. Thescaling can be designed such that the gap is big enough to separate theadjacent primitives to minimize the stitching issue and small enough tomake the reconstructed 3D object appear seamless. The predeterminedfactor can be determined based on information of the display, e.g., amaximum spatial resolution of the display.

In some cases, the scaling operation can be applied to primitive data ofa primitive obtained from the holographic render, e.g., the holographicrender 130 of FIG. 1A, and the scaled primitive data of the primitive issent to the Maxwell holographic controller, e.g., the controller 112 ofFIG. 1A. In some cases, the controller can perform the scaling operationon the primitive data obtained from the holographic render, beforecalculating EM contributions of the primitives to the display elementsof the display.

Exemplary Implementations for Texture Mapping

Texture mapping is a technique developed in computer graphics. The basicidea is to take a source image and apply it as a decal to a surface in aCGI system, enabling detail to be rendered into the scene without theneed for the addition of complex geometry. The texture mapping caninclude techniques for the creation of realistic lighting and surfacee□ects in the CGI system, and can refer universally to the applicationof surface data to triangular meshes.

In Maxwell holography, flat shaded and also interpolated triangularmeshes can be rendered in genuine 3D using the analytic relationshipbetween arbitrary triangles in space and a phase map on a holographicdevice. However, to be compatible with modern rendering engines, theability to map information on the surface of these triangles isinvolved. This can present a real problem, in that the speed of themethod is derived from the existence of the analytic mapping, which doesnot admit data-driven amplitude changes.

Discrete Cosine Transform (DCT) is an image compression technique andcan be considered as the real-valued version of the FFT (fast Fouriertransform). DCT depends on an encode-decode process that assigns weightsto cosine harmonics in a given image. The result of an encode is a setof weights equal in number to the number of pixels in the originalimage, and if every weight is used to reconstruct an image, there willbe no loss in information. However, in many images, acceptablereconstructions can be made from a small subset of the weights, enablinglarge compression ratios.

The decode (render) process of the DCT in two dimensions involves aweighted double sum over every DCT weight and every destination pixel.This can be applied to Maxwell holography for texture mapping. InMaxwell holography, triangle rendering involves a “spiked” doubleintegral, in phase space, to determine the phase contribution of anyindividual phasel to the triangle in question. The integral can befolded into a double sum which mirrors the one in the DCTreconstruction, and then re-derive the analytic triangle expression interms of the DCT weights. This implementation of DCT technique inMaxwell holographic calculations enables to draw full, texture mappedtriangles, to employ image compression to the data for the renderedtexture triangles, and to take advantage of existing toolsets thatautomatically compress texture and image data using DCT/JPEG.

In some implementations, to draw a Maxwell holographic texturedtriangle, a spatial resolution desired for the mapping on a specifiedsurface is first calculated. Then a texture with the resolution issupplied, and DCT compressed with angular and origin information tocorrectly orient it on the triangle is obtained. Then, the trianglecorners and a list of DCT weights are included in the index primitivelist and sent to the Maxwell holographic controller. The DCT weights canbe included in the EM contributions of the triangle primitive to eachdisplay element. The texture triangle can be n times slower than a flattriangle, where n is the number of (nonzero) DCT weights that are sentwith the primitive. Modern techniques for “fragment shading” can beimplemented in the Maxwell holographic system, with the step of the DCTencode replacing the filter step for traditional projective rendering.

As an example, the following expression shows the DCT weights B_(pq) foran image:

$\begin{matrix}{{B_{pq} \equiv {\sigma_{p}\sigma_{q}{\sum_{m = 0}^{M - 1}{\sum_{n = 0}^{N - 1}{A_{mn}{\cos \left\lbrack \frac{{\pi \left( {{2m} + 1} \right)}p}{2M} \right\rbrack}{\cos \left\lbrack \frac{{\pi \left( {{2n} + 1} \right)}q}{2N} \right\rbrack}\mspace{14mu} {where}}}}}}\mspace{20mu} {\sigma_{p} = \left\{ {\begin{matrix}{1\text{/}\sqrt{M}} & {p = 0} \\\sqrt{2\text{/}M} & {else}\end{matrix},{\sigma_{q} = \left\{ {\begin{matrix}{1\text{/}\sqrt{N}} & {q = 0} \\\sqrt{2\text{/}N} & {else}\end{matrix},} \right.}} \right.}} & {(5),}\end{matrix}$

M and N are corners of a rectangular image, and (p, q) is a DCT term.

By decoding, the amplitude value A_(mn) can be obtained as follows:

A _(mn)=Σ_(p=0) ^(M-1)Σ_(q=0) ^(N-1)σ_(p)σ_(q) A* _(mn)  (6),

where

$A_{mn}^{*} = {B_{pq}{\cos \left\lbrack \frac{{\pi \left( {{2m} + 1} \right)}p}{2M} \right\rbrack}{{\cos \left\lbrack \frac{{\pi \left( {{2n} + 1} \right)}q}{2N} \right\rbrack}.}}$

When calculating the EM contribution of the textured triangle primitiveto a display element (e.g., a phasel), a DCT term with a correspondingDCT weight A*_(mn) can be included in the calculation as follows:

φ_(pq)=Σ_(y=0) ^(Y)Σ_(x=0) ^(X) A* _(mn) T  (7),

where X, Y are corners of the triangle in the coordinate system, Tcorresponds to the EM contribution of the triangle primitive to thedisplay element, and φ_(pq) is the partial contribution for non-zeroterm B_(pq) in the DCT. The number of (p,q) DCT terms can be selected byconsidering both the information loss in reconstruction and theinformation compression.

Exemplary Process

FIG. 4 is a flowchart of an exemplary process 400 of displaying anobject in 3D. The process 400 can be performed by a controller for adisplay. The controller can be the controller 112 of FIG. 1A or 152 ofFIG. 1B. The display can the display 114 of FIG. 1A or 156 of FIG. 1B.

Data including respective primitive data for primitives corresponding toan object in a 3D space is obtained (402). The data can be obtained froma computing device, e.g., the computing device 102 of FIG. 1A. Thecomputing device can process a scene to generate the primitivescorresponding to the object. The computing device can include a rendererto generate the primitive data for the primitives. In someimplementations, the controller generates the data itself, e.g., byrendering the scene.

The primitives can include at least one of a point primitive, a lineprimitive, or a polygon primitive. The list of primitives is indexed ina particular order, e.g., by which the object can be reconstructed. Theprimitive data can include color information that has at least one of atextured color or a gradient color. For example, the line primitive canhave at least one of a gradient color or as a textured color. Thepolygon primitive can also have at least one of a gradient color or atextured color. The primitive data can also include texture informationof the primitive and/or shading information on one or more surfaces ofthe primitive (e.g., a triangle). The shading information can include amodulation on at least one of color or brightness on the one or moresurfaces of the primitive. The primitive data can also includerespective coordinate information of the primitive in the 3D coordinatesystem.

The display can include a number of display elements, and the controllercan include a number of computing units. Respective coordinateinformation of each of the display elements in the 3D coordinate systemcan be determined based on the respective coordinate information of thelist of primitives in the 3D coordinate system. For example, a distancebetween the display and the object corresponding to the primitives canbe predetermined. Based on the predetermined distance and the coordinateinformation of the primitives, the coordinate information of the displayelements can be determined. The respective coordinate information ofeach of the display elements can correspond to a logical memory addressfor the element stored in a memory. In such a way, when the controllerloops in a logical memory address for a display element in a logicalmemory space of the controller, a corresponding actual physical locationfor the display element in the space can be identified.

An EM field contribution from each of the primitives to each of thedisplay elements is determined by calculating EM field propagation fromthe primitive to the element in the 3D coordinate system (404). The EMfield contribution can include at least one of a phase contribution oran amplitude contribution.

As illustrated above with respect to FIGS. 3A-3C, at least one distancebetween the primitive and the display element can be determined based onthe respective coordinate information of the display element and therespective coordinate information of the primitive. In some cases, foreach primitive, the at least one distance can be calculated or computedjust once. For example, the controller can determine a first distancebetween a first primitive of the primitives and a first element of thedisplay elements based on the respective coordinate information of thefirst primitive and the respective coordinate information of the firstelement and determining a second distance between the first primitiveand a second element of the elements based on the first distance and adistance between the first element and the second element. The distancebetween the first element and the second element can be predeterminedbased on a pitch of the plurality of elements of the display.

The controller can determine the EM field contribution to the displayelement from the primitive based on a predetermined expression for theprimitive and the at least one distance. In some cases, as illustratedabove with respect to FIGS. 3A-3C, the predetermined expression can bedetermined by analytically calculating the EM field propagation from theprimitive to the element. In some cases, the predetermined expression isdetermined by solving Maxwell's equations. Particularly, the Maxwell'sequations can be solved by providing a boundary condition defined at asurface of the display. The boundary condition can include a Dirichletboundary condition or a Cauchy boundary condition. The primitives andthe display elements are in the 3D space, and the surface of the displayforms a portion of a boundary surface of the 3D space. The predeterminedexpression can include at least one of functions that include a sinefunction, a cosine function, and an exponential function. Duringcomputation, the controller can identify a value of the at least one ofthe functions in a table stored in a memory, which can improve acomputation speed. The controller can determine the EM fieldcontribution to each of the display elements for each of the primitivesby determining a first EM field contribution from a first primitive to adisplay element in parallel with determining a second EM fieldcontribution from a second primitive to the display element.

For each of the display elements, a sum of the EM field contributionsfrom the list of primitives to the display element is generated (406).

In some implementations, the controller determines first EM fieldcontributions from the primitives to a first display element and sumsthe first EM field contributions for the first element and determiningsecond EM field contributions from the primitives to a second displayelement and sums the second EM field contributions for the seconddisplay element. The controller can include a number of computing units.The controller can determine an EM field contribution from a firstprimitive to the first element by a first computing unit in parallelwith determining an EM field contribution from a second primitive to thefirst element by a second computing unit.

In some implementations, the controller determines first respective EMfield contributions from a first primitive to each of the displayelements and determine second respective EM field contributions from asecond primitive to each of the display elements. Then the controlleraccumulates the EM field contributions for the display element by addingthe second respective EM field contribution to the first respective EMfield contribution for the display element. Particularly, the controllercan determine the first respective EM field contributions from the firstprimitive to each of the display elements by using a first computingunit in parallel with determining the second respective EM fieldcontributions from the second primitive to each of the display elementsby using a second computing unit.

A first control signal is transmitted to the display, the first controlsignal being for modulating at least one property of each displayelement based on the sum of the field distributions to the displayelement (408). The at least one property of the element includes atleast one of a refractive index, an amplitude index, a birefringence, ora retardance.

The controller can generate, for each of the display elements, arespective control signal based on the sum of the EM field contributionsfrom the primitives to the element. The respective control signal is formodulating the at least one property of the element based on the sum ofthe EM field contributions from the primitives to the element. That is,the first control signal includes the respective control signals for thedisplay elements.

In some examples, the display is controlled by electrical signals. Thenthe respective control signal can be an electrical signal. For example,an LCOS display includes an array of tiny electrodes whose voltage isindividually controlled as element intensities. The LCOS display can befilled with a birefringent liquid crystal (LC) formulation that changesits refractive index. Thus, the respective control signals from thecontroller can control the relative refractive index across the displayelements accordingly the relative phase of light passing through thedisplay.

As discussed above, the display surface forms a part of the boundarysurface. The controller can multiple a scale factor to the sum of thefield contributions for each of the elements to obtain a scaled sum ofthe field contributions, and generate the respective control signalbased on the scaled sum of the field contributions for the element. Insome cases, the controller can normalize the sum of the fieldcontributions for each of the elements, e.g., among all the elements,and generate the respective control signal based on the normalized sumof the field contributions for the element.

A second control signal is transmitted to an illuminator, the secondcontrol signal for turning on the illuminator to illuminate light on themodulated display (410). The controller can generate and transmit thesecond control signal in response to determining a completion ofobtaining the sum of the field contributions for each of the displayelements. Due to time symmetry (or conservation of energy), themodulated elements of the display can cause the light to propagate indifferent directions to form a volumetric light field corresponding tothe object in the 3D space. The volumetric light field can correspond toa solution of Maxwell's equations with a boundary condition defined bythe modulated elements of the display.

In some implementations, the illuminator is coupled to the controllerthrough a memory buffer configured to control amplitude or brightness ofone or more light emitting elements in the illuminator. The memorybuffer for the illuminator can have a smaller size than a memory bufferfor the display. A number of the light emitting elements in theilluminator can be smaller than a number of the elements of the display.The controller can be configured to activate the one or more lightemitting elements of the illuminator simultaneously.

In some examples, the illuminator includes two or more light emittingelements each configured to emit light with a different color. Thecontroller can be configured to sequentially modulate the display withinformation associated with a first color during a first time period andmodulate the display with information associated with a second colorduring a second, sequential time period, and to control the illuminatorto sequentially turn on a first light emitting element to emit lightwith the first color during the first time period and a second lightemitting element to emit light with the second color during the secondtime period. In such a way, a multi-color object can be displayed in the3D space.

In some examples, the display has a resolution small enough to diffractlight. The illuminator can emit a white light on the display which candiffract the white light into light with different colors to therebydisplay a multi-color object.

Exemplary Systems

FIGS. 5A-5F show implementations of example systems for 3D displays. Anyone of systems can correspond to, for example, the system 100 of FIG.1A.

FIG. 5A illustrates a system 500 with a reflective display. The system500 includes a computer 502, a controller 510 (e.g., ASIC), a display512 (e.g., an LCOS device), and an illuminator 514. The computer 502 canbe the computing device 102 of FIG. 1A, the controller 510 can be thecontroller 112 of FIG. 1A, the display 512 can be the display 114 ofFIG. 1A, and the illuminator 514 can be the illuminator 116 of FIG. 1A.

As illustrated in FIG. 5A, the computer 502 includes an application 504that has a renderer 503 for rendering a scene of an object. The renderedscene data is sequentially processed by a video driver 505 and a GPU506. The GPU 506 can be the GPU 108 of FIG. 1A and configured togenerate a list of primitives corresponding to the scene and respectiveprimitive data. For example, the video driver 505 can be configured toprocess the rendered scene data and generate a list of primitives. Asnoted above, the GPU 506 can include a conventional 2D renderer, e.g.,the conventional 2D renderer 120 of FIG. 1A, to render the primitivesinto a list of items to draw on a 2D display screen 508. The GPU 506 orthe controller 510 can include a holographic renderer, e.g., theholographic renderer 130 of FIG. 1A, to render the list of primitivesinto graphic data to be displayed by the display 512.

The controller 510 is configured to receive the graphic data from thecomputer 502, compute EM field contributions from the list of primitivesto each of elements of the display 512, and generate a respective sum ofthe EM field contributions from the primitives to each of the elements.The controller 510 can generate respective control signals to each ofthe display elements for modulating at least one property of the displayelement. The controller can transmit the respective control signals tothe display elements of the display 512 through a memory buffer 511 forthe display 512.

The controller 510 can also generate and transmit a control signal,e.g., an illumination timing signal, to activate the illuminator 514.For example, the controller 510 can generate and transmit the controlsignal in response to determining that the computations of the sums ofEM field contributions from the primitives to the display elements arecompleted. As noted above, the controller 510 can transmit the controlsignal to the illuminator 514 via a memory buffer. The memory buffer canbe configured to control amplitude or brightness of light emittingelements in the illuminator 514 and activate the light emitting elementssimultaneously.

As illustrated in FIG. 5A, the illuminator 514 can emit a collimatedlight beam 516 that is incident on a front surface of the display 512 atan incident angle in a range between 0 degree and 90 degree. The emittedlight beam is reflected from the front surface of the display 512 toform a holographic light field 518, corresponding to the object, whichcan be seen by a viewer.

FIG. 5B illustrates another system 520 with another reflective display524. Compared to the system 500 of FIG. 5A, the system 520 has a largerreflective display 524. To accommodate this, a display controller 522 isincluded in a wedge housing that can be a support for an illuminator526. The controller 522 is similar to the controller 510 of FIG. 5A andcan be configured to receive graphic data from a computer 521, computeEM field contributions from primitives to each of display elements ofthe display 524, and generate a respective sum of the EM fieldcontributions from the primitives to each of the display elements. Thecontroller 522 then generates respective control signals to each of thedisplay elements for modulating at least one property of the displayelement and transmits the respective control signals to the displayelements of the display 524 through a memory buffer 523 for the display524.

The controller 522 also transmits a control signal to the illuminator526 to activate the illuminator 526. The illuminator 526 emits adivergent or semi-collimated light beam 527 to cover a whole surface ofthe display 524. The light beam 524 is reflected by the modulateddisplay 524 to form a holographic light field 528.

FIG. 5C illustrates a system 530 with a transmissive display 534. Thetransmissive display 534, for example, can be a large scale display. Thesystem 530 includes a controller 532 which can be similar to thecontroller 510 of FIG. 5A. The controller 532 can be configured toreceive graphic data from a computer 531, compute EM field contributionsfrom primitives to each of display elements of the display 534, andgenerate a respective sum of the EM field contributions from theprimitives to each of the display elements. The controller 532 thengenerates respective control signals to each of the display elements formodulating at least one property of the display element and transmitsthe respective control signals to the display elements of the display534 through a memory buffer 533 for the display 534.

The controller 532 also transmits a control signal to an illuminator 536to activate the illuminator 536. Different from the system 500 of FIG.5A and the system 520 of FIG. 5B, the illuminator 536 in the system 530is positioned behind a rear surface of the display 534. To cover a largesurface of the display 534, the illuminator 536 emits a divergent orsemi-collimated light beam 535 on the rear surface of the display 534.The light beam 524 is transmitted through the modulated display 534 toform a holographic light field 538.

FIG. 5D illustrates another system 540 with a transmissive display 544.The system 540 also includes a controller 542 and an illuminator 546.The controller 542 can be similar to the controller 510 of FIG. 5A, andcan be configured to receive graphic data from a computer 541, performcomputation on the graphic data, generate and transmit control signalsfor modulation to the display 544 and a timing signal to activate theilluminator 546.

The illuminator 546 can include a light source 545 and a waveguide 547.Light emitted from the light source 545 can be coupled to the waveguide547, e.g., from a side cross-section of the waveguide. The waveguide 547is configured to guide the light to illuminate a surface of the display544 uniformly. The light guided by the waveguide 547 is incident on arear surface of the display 544 and transmitted through the display 544to form a holographic light field 548.

Different from the system 500 of FIG. 5A, 520 of FIG. 5B, 530 of FIG.5C, in the system 540, the controller 542, the display 544, and thewaveguide 547 are integrated together into a single unit 550. In somecases, the waveguide 547 and the light source 545 can be integrated asan active waveguide illuminator in a planar form, which can furtherincrease a degree of integration of the single unit 550. As discussedabove, the single unit 500 can be connected with other similar units 550to form a larger holographic display device.

FIG. 5E illustrates another system 560 with another transmissive display564. Compared to the system 540, the transmissive display 564 canpotentially implement a display that is larger than the transmissivedisplay 544. For example, the transmissive display 564 can have a largerarea than a controller 562, and to accommodate this, the controller 562can be positioned away from the display 564. The system 560 includes anilluminator 566 that has a light source 565 and a waveguide 567. Thewaveguide 567 is integrated with the display 564, e.g., to a rearsurface of the display 564. In some implementations, the display 564 isfabricated on a front side of a substrate and the waveguide 567 can befabricated on a back side of the substrate.

The controller 562 can be similar to the controller 510 of FIG. 1A andconfigured to receive graphic data from a computer 561, performcomputation on the graphic data, generate and transmit control signalsfor modulation to the display 564 through a memory buffer 563 and atiming signal to activate the light source 565. Light emitted from thelight source 565 is guided in the waveguide 567 to illuminate on therear surface of the display 564 and transmitted through the display 564to form a holographic light field 568.

FIG. 5F illustrates another system 570 with a reflective display 574.The reflective display 574, for example, can be a large display. Awaveguide 577 of an illuminator 576 is positioned on a front surface ofthe reflective display 574. A controller 572, similar to the controller562 of FIG. 5E, can be configured to receive graphic data from acomputer 571, perform computation on the graphic data, generate andtransmit control signals for modulation to the display 574 through amemory buffer 573 and a timing signal to activate a light source 575 ofthe illuminator 576. Light coupled from a light source 575 of theilluminator 576 is guided to be incident on the front surface of thedisplay 574 and reflected by the front surface to form a holographiclight field 578.

Exemplary Display Implementations

As noted above, a display in Maxwell holography can be a phasemodulating device. A phase element of the display (or a display element)can be represented as a phasel. For illustration only, a liquid crystalon silicon (LCOS) device is discussed below to function as the phasemodulating device. The LCOS device is a display using a liquid crystal(LC) layer on top of a silicon backplane. The LCOS device can beoptimized to achieve minimum possible phasel pitch, minimum cross-talkbetween phasels, and/or a large available phase modulation or retardance(e.g., at least 2π).

A list of parameters can be controlled to optimize the performance ofthe LCOS device, including a birefringence of LC mixture (Δn), a cellgap (d), dielectric anisotropy of LC mixture (Δε), rotational viscosityof the LC mixture (η), maximum applied voltage between the siliconbackplane and a common electrode on top of the LC layer (V).

There can be a fundamental trade-off that exists between parameters ofthe liquid crystal material. For example, a fundamental boundingparameter is the available phase modulation or retardance (Re), whichcan be expressed as:

Re=4π·Δn·d/λ  (8),

where λ is the wavelength of an input light. If the retardance Re needsto be at least 27 for a red light with a wavelength of about 0.633 μm,then

Δn·d≥0.317 μm  (9).

The above expression implies that there is a direct trade-off betweencell gap (d) and birefringence (Δn) of the LC mixture.

Another bounding parameter is the switching speed, or the switching time(T) it takes for the liquid crystal (LC) molecules in an LC layer toreach the desired orientation after a voltage is applied. For example,for real-time video (˜60 Hz) using 3-color field sequential colorsystem, a minimum of 180 Hz modulation of the LC layer is involved,which puts an upper bound on the LC switching speed of 5.6 milliseconds(ms). Switching time (T) is related to a number of parameters includingthe liquid crystal, cell gap, and applied voltage. First, T isproportional to d². As the cell gap d is decreased, the switching timedecreases as the square. Second, the switching time is also related tothe dielectric anisotropy (Δε) of the liquid crystal (LC) mixture, witha higher dielectric anisotropy resulting in a shorter switching time anda lower viscosity also resulting in a shorter switching time.

A third bounding parameter can be the fringing field. Due to the highelectron mobility of crystalline silicon, an LCOS device can befabricated with a very small phasel size (e.g., less than 10 μm) andsubmicron inter-phasel gap. When the adjacent phasels are operated atdifferent voltages, the LC directors near the phasel edges are distortedby the lateral component of the fringing field, which significantlydegrades the electro-optic performance of the device. In addition, asthe phasel gap becomes comparable to the incident light wavelength,diffraction effect can cause severe light loss. The phasel gap needs tobe kept at less than or equal to a phasel pitch to keep noise within anacceptable level.

In some examples, the LCOS device is designed to have a phasel pitch of2 μm and a cell gap of approximately 2 μm as well if the fringe fieldbounding condition is observed. According to the above expressionΔn·d≥0.317 μm, Δn needs to be equal to 0.1585 or greater, which isachievable using current liquid crystal technology. Once the minimumbirefringence for a given phasel pitch is determined, the LC can beoptimized for switching speed, e.g., by increasing the dielectricanisotropy and/or decreasing the rotational viscosity.

Nonuniform Phasels Implementations for Displays

In an LCOS device, a circuit chip, e.g., a complementarymetal-oxide-semiconductor (CMOS) chip or equivalent, controls thevoltage on reflective metal electrodes buried below the chip surface,each controlling one phasel. A common electrode for all the phasels issupplied by a transparent conductive layer made of indium tin oxide onthe cover glass. The phasels can have identical sizes and same shape(e.g., square). For example, a chip can have 1024×768 plates, each withan independently addressable voltage. As noted above, when the phaselgap becomes comparable to the incident light wavelength, diffractioneffect can appear in the periodic structure of the LCOS device, whichmay cause severe light loss.

In Maxwell holographic calculations, each phasel receives a sum of EMcontributions from each primitive and is relatively independent fromeach other. Thus, the phasels of the LCOS device in Maxwell holographycan be designed to be different from each other. For example, asillustrated in FIG. 6A, the LCOS device 600 can be made of a number ofnonuniform (or irregular) phasels 602. At least two phasels 602 havedifferent shapes. The nonuniform shapes of the phasels 602 can greatlyreduce or eliminate diffractive aberrations, among other effects andthus improve image quality. Although the phasels can have nonuniformshapes, the phasels can be designed to have a size distribution (e.g.,about 3 μm) that satisfies a desired spatial resolution. The siliconbackplane can be configured to provide a respective circuit (e.g.,including a metal electrode) for each of the phasels according to theshape of the phasel.

In an array of phasels in an LCOS device, to select a specific phasel, afirst voltage is applied to a word line connecting a row of phaselsincluding the specific phasel and a second voltage is applied to a bitline connecting a column of phasels including the specific phasel. Aseach phasel has a resistance, the operational speed of the LCOS devicecan be limited.

As noted above, in Maxwell holography, the phasels can have differentsizes. As illustrated in FIG. 6B, an LCOS device 650 is designed to haveone or more phasels 654 having a size larger than the other phasels 652.All of the phasels can still have a size distribution that satisfies thedesired resolution. For example, 99% of the phasels have a size of 3 μm,and only 1% of the phasels have a size of 6 μm. The larger size of thephasel 654 allows to arrange at least one buffer 660 in the phasel 654besides other circuitry same as in the phasel 652. The buffer 660 isconfigured to buffer the applied voltage such that the voltage is onlyapplied to a smaller number of phasels within a row or column ofphasels. The buffer 660 can be an analog circuit, e.g., made of atransistor, or a digital circuit, e.g., made of a number of logic gates,or any combination thereof.

For example, as illustrated in FIG. 6B, a voltage is applied to a wordline 651 and another voltage is applied to a bit line 653 to select aparticular phasel 652*. The phasel 652* is in the same row as the largerphasel 654 including the buffer 660. The voltage is mainly applied tothe first number of phasels in the row and before the larger phasel 654and obstructed by the buffer 660 in the larger phasel 654. In such away, the operational speed of the LCOS device 650 can be improved. Withthe larger size of the phasels 654, other circuitry can be also arrangedin the LCOS device 650 to further improve the performance of the LCOSdevice 650. Although the phasels 654 and the phasels 652 in FIG. 6B havesquare shape, the phasels can also have different shapes as illustratedin FIG. 6A as long as there are one or more phasels 654 having a largersize than the other phasels 652.

Exemplary Calibrations

The unique nature of Maxwell holography in the present disclosure allowsfor the protection of calibration techniques that can create asignificant competitive advantage in the actual production of highquality displays. A number of calibration techniques can be implementedto be combined with the Maxwell holographic computational techniques,including:

(i) using image sensors in conjunction with a Dirichlet boundarycondition modulator and/or in conjunction with mechanical and softwarediffractive and non-diffractive calibration techniques;

(ii) software alignments and software calibrations including individualcolor calibrations and alignments with Dirichlet boundary conditionmodulators; and

(iii) embedding silicon features in the boundary condition modulatorsthat allow for a photo detection to be built directly into the modulatorthat when combined with Maxwell holography create a powerful and uniqueapproach to simplifying manufacturing calibration processes.

In the following, for illustration only, three types of calibrations areimplemented for phase based displays, e.g., LCOS displays. Each phaseelement can be represented as a phasel.

Phase Calibration

An amount of phase added to light impinging upon an LCOS phase element(or phasel) can be known directly by a voltage applied to the LCOSphasel. This is due to the birefringent liquid crystal (LC) rotating inthe presence of an electric field and thus changing its index ofrefraction and slowing down light to alter its phase. The altered phasecan depend upon electrical characteristics of the liquid crystal (LC)and the silicon device in which the LC resides. Digital signals sent tothe LCOS need to be transformed into correct analog voltages to achievehigh quality holographic images. Phase calibration is involved for theLCOS device to ensure that a digital signal is properly transformed intoan analog signal applied to the LC such that it produces the greatestamount of phase range. This conversion is expected to result in a linearbehavior. That is, as the voltage is changed by fixed increments, thephase also changes by a fixed increment, regardless of the startingvoltage value.

In some cases, an LCOS device allows a user to alter a digital-to-analogconverter (DAC) such that the user has a control over the amount ofanalog voltage output given a digital input signal. A digitalpotentiometer can be applied to each input bit. For example, if thereare 8 input bits, there can be 8 digital potentiometers corresponding toeach input bit. The same digital inputs from the digital potentiometerscan be applied to all phasels of the LCOS device. Bits set to “1”activate a voltage, and bits set to “0” do not activate the voltage. Allvoltages from such “1” bits are summed together to obtain the finalvoltage sent to each phasel. There may also be a DC voltage applied inall cases such that all “0” bits results in a baseline non-zero voltage.Thus, the phase calibration of the LCOS device can be implemented bysetting values of the digital potentiometers for the LCOS device. Forexample, as noted above, a controller can compute EM field contributionsfrom a list of primitives to each of phasels of a display, generate arespective sum of the EM field contributions from the primitives to eachof the phasels, and generate respective control signals to each of thephasels for modulating a phase of the phasel. The same digital inputsfrom the digital potentiometers can be applied to adjust the respectivecontrol signals to all of the phasels of the LCOS device, which isdifference from a phasel-by-phasel based phase calibration. The digitalinputs can be set once for a duration of an operation of the LCOSdevice, e.g., for displaying a hologram.

To determine an optimal set of phase calibration values for the digitalinputs, a genetic algorithm can be applied, where there are many inputvalues that lead to one output value, such as phase range or holographicimage contrast. This output value can be reduced to one number known asthe fitness. The genetic algorithm can be configured to exploredifferent combinations of input values until it achieves an output withthe highest fitness. In some cases, the algorithm can take two or moreof the most fit inputs and combine a number of their constituent valuestogether to create a new input that has characteristics of the takeninputs but is different from each of the taken inputs. In some cases,the algorithm can alter one of these constituent values to something notfrom either of the taken fit inputs, which is represented as a“mutation” and can add a variety to the available fit inputs. In somecases, one or more optimal values can be found by taking advantage ofthe knowledge gained from prior measurements with good results whiletrying new values so the optimal values do not be restricted to a localmaximum.

There can be multiple ways to calculate the fitness output value. Oneway is to calculate the phase change of the light given a set of digitalinputs applied to all the phasels on the LCOS. In this scheme, theincident light can be polarized. Upon impinging upon the LCOS, theincident light's polarization can change depending on the rotation ofthe LC. The incident light can be reflected back through anotherpolarizer set to either the same polarization or 90 degrees differentfrom the original polarization and then into a light detector.Therefore, when the LC rotation changes, the intensity as viewed fromthe light detector can change. Accordingly, the phase change of thelight can be perceived indirectly through the intensity variations.Another way to calculate the phase change is to measure the intensitydifference of a Maxwell holographic reconstruction from the background.This is most effective in a projective display. Measuring the intensityin such an instance may need the use of computer vision algorithms toidentify the Maxwell holographic reconstruction and measure itsintensity.

Alignment Calibration

Light sources are not guaranteed to be aligned within a holographicdevice and therefore need to be aligned. Different liquid crystals (LC)can also behave differently given the wavelengths of the light sources.Moreover, both the LC and light sources can change device to device,giving different characteristics, e.g., object scaling, to the sameinput hologram when shown in a different base color. Furthermore,certain hardware features can apply different optical effects to theoutput light, e.g., lensing, that also need correction.

In some implementations, the problems described above can be addressedby applying a mathematical transform to a phase calculated for a phaselof a display. The phase is a respective sum of the EM fieldcontributions from a list of primitives to the phasel. The mathematicaltransform can be derived from a mathematical expression, e.g., a Zernikepolynomial, and can be varied by altering polynomial coefficients orother varying input values. The mathematical transform can varyphasel-by-phasel as well as by color. For example, there is a Zernikepolynomial coefficient that corresponds to the amount of tilt to beapplied to the light after it reflects off of the display.

To determine these coefficients/input values, a hardware setup can becreated where a camera is pointed at a reflective surface in the case ofa projective display and directly into the LCOS in the case of adirect-view display. A series of holographic test patterns and objectscan be sent to the display and are viewed by the camera. The camera canuse machine vision algorithms to determine what is being displayed andthen calculate its fitness. For example, if a grid of dots is the testpattern, then the fitness is how close they are together, how centeredthey are positioned, how much distortion they have (e.g., scale orpincushion), etc. There can be different fitness values for differentcharacteristics. Depending on these values, corrections can be applied,e.g., in the form of changing coefficients to the Zernike polynomial,until the fitness reaches a predetermined satisfactory level. These testpatterns can be rendered at different distances to ensure that alignmentis consistent for objects at all the distances, and not just one pointin particular. Such depth-based calibrations involve iterative processesthat involve altering the depth of the holographic test pattern as wellas the reflective surface in the case of a projective display where theprevious calibrations may be repeated until converging upon a solutionthat works at both depths. Finally, white dots can be displayed to showthe effectiveness of the calibration.

Color Calibration

In displays, holographic or otherwise, it is important that, when anytwo units are rendering the same image, colors match between displaysand additionally match colors defined by television (TV) and computerdisplay standards, like the Rec.709 standard for high-definitiontelevision (HDTV) or the sRGB color space of computer monitors.Different batches of hardware components, e.g., LEDs and laser diodes,can exhibit different behaviors for the same inputs and can outputdifferent colors when perceived by the human eye. Therefore, it isimportant to have a color standard to which all display units can becalibrated.

In some implementations, an objective measurement of color specified bymeasurements of intensity and chromaticity can be obtained by measuringcolor intensity against a Commission internationale de l′éclairage (CIE)Standard Observer curves. By requesting that each display reproduces asample set of known colors and intensities, then measuring the outputlight using a Colorimeter device calibrated to the CIE Standard Observercurves, the color output of a device in CIE XYZ color space can beobjectively defined. Any deviation of the measured values from the knowngood values can be used to adapt the output colors on the display tobring it back into alignment, which can be implemented using aniterative measure-adapt-measure feedback loop. Once a Maxwellholographic device produces accurate outputs for a given set of inputs,the final adaptations can be encoded as look-up tables for theilluminators that map input values to output intensities, and colormatrix transformations that transform input colors to output color spacevalues. These calibration tables can be embedded in the device itself toproduce reliable objective output colors.

Additionally, given an LCOS device with fine enough features to controldiffraction with sub-wavelength accuracy, there may be no need fortri-stimulus illumination (e.g., linear mixes of red, green, and blue),and the LCOS device can be illuminated with a single wide spectrum lightsource and selectively tune the phasels output to produce tri-, quad-,even N-stimulus output colors which, combined with spatial ditheringpatterns, can reproduce a complete spectral output of a color ratherthan the common tri-stimulus approximation. Given a sufficiently widespectrum illuminator this allows Maxwell holography to produce anyreflected color that lies inside the spectral locus of human visualsystem.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Implementations of the subject matter described inthis specification can be implemented as one or more computer programs,such as, one or more modules of computer program instructions encoded ona tangible, non-transitory computer-storage medium for execution by, orto control the operation of, data processing apparatus. Alternatively orin addition, the program instructions can be encoded on an artificiallygenerated propagated signal, such as, a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. The computer-storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The terms “data processing apparatus,” “computer,” or “electroniccomputer device” (or equivalent as understood by one of ordinary skillin the art) refer to data processing hardware and encompass all kinds ofapparatus, devices, and machines for processing data, including by wayof example, a programmable processor, a computer, or multiple processorsor computers. The apparatus can also be or further include specialpurpose logic circuitry, for example, a central processing unit (CPU),an FPGA (field programmable gate array), or an ASIC(application-specific integrated circuit). In some implementations, thedata processing apparatus and special purpose logic circuitry may behardware-based and software-based. The apparatus can optionally includecode that creates an execution environment for computer programs, forexample, code that constitutes processor firmware, a protocol stack, adatabase management system, an operating system, or a combination of oneor more of them. The present specification contemplates the use of dataprocessing apparatuses with or without conventional operating systems.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, for example,one or more scripts stored in a markup language document, in a singlefile dedicated to the program in question, or in multiple coordinatedfiles, for example, files that store one or more modules, sub-programs,or portions of code. A computer program can be deployed to be executedon one computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork. While portions of the programs illustrated in the variousfigures are shown as individual modules that implement the variousfeatures and functionality through various objects, methods, or otherprocesses, the programs may instead include a number of sub-modules,third-party services, components, libraries, and such, as appropriate.Conversely, the features and functionality of various components can becombined into single components as appropriate.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, such as, a CPU, a GPU, an FPGA, or an ASIC.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors, both, or any other kindof CPU. Generally, a CPU will receive instructions and data from aread-only memory (ROM) or a random access memory (RAM) or both. The mainelements of a computer are a CPU for performing or executinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto, receive data from or transfer data to, or both, one or more massstorage devices for storing data, for example, magnetic, magneto-opticaldisks, or optical disks. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, for example, amobile telephone, a personal digital assistant (PDA), a mobile audio orvideo player, a game console, a global positioning system (GPS)receiver, or a portable storage device, for example, a universal serialbus (USB) flash drive, to name just a few.

Computer readable media (transitory or non-transitory, as appropriate)suitable for storing computer program instructions and data include allforms of non-volatile memory, media and memory devices, including by wayof example semiconductor memory devices, for example, erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), and flash memory devices;magnetic disks, for example, internal hard disks or removable disks;magneto-optical disks; and CD-ROM, DVD-R, DVD-RAM, and DVD-ROM disks.The memory may store various objects or data, including caches, classes,frameworks, applications, backup data, jobs, web pages, web pagetemplates, database tables, repositories storing business and dynamicinformation, and any other appropriate information including anyparameters, variables, algorithms, instructions, rules, constraints, orreferences thereto. Additionally, the memory may include any otherappropriate data, such as logs, policies, security or access data,reporting files, as well as others. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, for example, a cathode ray tube (CRT), liquidcrystal display (LCD), light emitting diode (LED), or plasma monitor,for displaying information to the user and a keyboard and a pointingdevice, for example, a mouse, trackball, or trackpad by which the usercan provide input to the computer. Input may also be provided to thecomputer using a touchscreen, such as a tablet computer surface withpressure sensitivity, a multi-touch screen using capacitive or electricsensing, or other type of touchscreen. Other kinds of devices can beused to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, forexample, visual feedback, auditory feedback, or tactile feedback; andinput from the user can be received in any form, including acoustic,speech, or tactile input. In addition, a computer can interact with auser by sending documents to and receiving documents from a device thatis used by the user; for example, by sending web pages to a web browseron a user's client device in response to requests received from the webbrowser.

The term “graphical user interface,” or “GUI,” may be used in thesingular or the plural to describe one or more graphical user interfacesand each of the displays of a particular graphical user interface.Therefore, a GUI may represent any graphical user interface, includingbut not limited to, a web browser, a touch screen, or a command lineinterface (CLI) that processes information and efficiently presents theinformation results to the user. In general, a GUI may include multipleuser interface (UI) elements, some or all associated with a web browser,such as interactive fields, pull-down lists, and buttons operable by thebusiness suite user. These and other UI elements may be related to orrepresent the functions of the web browser.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, for example, as a data server, or that includes a middlewarecomponent, for example, an application server, or that includes afront-end component, for example, a client computer having a graphicaluser interface or a web browser through which a user can interact withan implementation of the subject matter described in this specification,or any combination of one or more such back-end, middleware, orfront-end components. The components of the system can be interconnectedby any form or medium of wireline or wireless digital datacommunication, for example, a communication network. Examples ofcommunication networks include a local area network (LAN), a radioaccess network (RAN), a metropolitan area network (MAN), a wide areanetwork (WAN), worldwide interoperability for microwave access (WIMAX),a wireless local area network (WLAN) using, for example, 902.11 a/b/g/nand 902.20, all or a portion of the Internet, and any othercommunication system or systems at one or more locations. The networkmay communicate with, for example, internet protocol (IP) packets, framerelay frames, asynchronous transfer mode (ATM) cells, voice, video,data, or other suitable information between network addresses.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In some implementations, any or all of the components of the computingsystem, both hardware and software, may interface with each other or theinterface using an application programming interface (API) or a servicelayer. The API may include specifications for routines, data structures,and object classes. The API may be either computer language-independentor -dependent and refer to a complete interface, a single function, oreven a set of APIs. The service layer provides software services to thecomputing system. The functionality of the various components of thecomputing system may be accessible for all service consumers via thisservice layer. Software services provide reusable, defined businessfunctionalities through a defined interface. For example, the interfacemay be software written in any suitable language providing data in anysuitable format. The API and service layer may be an integral or astand-alone component in relation to other components of the computingsystem. Moreover, any or all parts of the service layer may beimplemented as child or sub-modules of another software module,enterprise application, or hardware module without departing from thescope of this specification.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particularimplementations of particular inventions. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable sub-combination. Moreover,although features may be described as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a sub-combination or variation ofa sub-combination.

Particular implementations of the subject matter have been described.Other implementations, alterations, and permutations of the describedimplementations are within the scope of the following claims as will beapparent to those skilled in the art. While operations are depicted inthe drawings or claims in a particular order, this should not beunderstood as requiring that such operations be performed in theparticular order shown or in sequential order, or that all illustratedoperations be performed (some operations may be considered optional), toachieve desirable results. In certain circumstances, multitasking orparallel processing may be advantageous and performed as deemedappropriate.

Accordingly, the earlier provided description of example implementationsdoes not define or constrain this specification. Other changes,substitutions, and alterations are also possible without departing fromthe spirit and scope of this specification.

1-185. (canceled)
 186. A system, comprising: a computing deviceconfigured to generate data comprising respective primitive data of aplurality of primitives corresponding to an object in athree-dimensional (3D) space; and a display system configured to receivethe graphic data from the computing device and process the graphic datafor presenting the object in the 3D space, the display systemcomprising: a display comprising a plurality of elements; and acontroller coupled to the display and configured to perform operationscomprising: for each of a plurality of primitives corresponding to anobject in the 3D space, determining an electromagnetic (EM) fieldcontribution to each of a plurality of elements of a display bycomputing, in a 3D coordinate system, EM field propagation from theprimitive to the element; and for each of the plurality of elements,generating a sum of the EM field contributions from the plurality ofprimitives to the element.
 187. The system of claim 186, wherein thecomputing device comprises an application programming interface (API)configured to create the primitives with the respective primitive databy rendering a computer generated (CG) model of the object.
 188. Thesystem of claim 186, wherein the controller comprises a plurality ofcomputing units, each of the computing units being configured to performrespective operations on one or more primitives of a plurality ofprimitives correspond to an object in a three-dimensional (3D) space.189. The system of claim 188, wherein the controller is locally coupledto the display, and wherein each of the computing units is coupled toone or more respective elements of the display and configured totransmit a respective control signal to each of the one or morerespective elements.
 190. The system of claim 188, wherein the computingunits are configured to operate in parallel.
 191. The system of claim1861, wherein the controller comprises at least one member selected fromthe group consisting of an application-specific integrated circuit(ASIC), a field-programmable gate array (FPGA), a programmable gatearray (PGA), a central processing unit (CPU), a graphics processing unit(GPU), and standard computing cells.
 192. The system of claim 186,wherein the display comprises a spatial light modulator (SLM) includinga digital micro-mirror device (DMD) or a liquid crystal on silicon(LCOS) device.
 193. The system of claim 186, wherein the display isconfigured to be phase modulated, amplitude modulated, or phase andamplitude modulated.
 194. The system of claim 186, wherein thecontroller is coupled to the display through a memory buffer.
 195. Thesystem of claim 186, further comprising an illuminator arranged adjacentto the display and configured to emit light on the display.
 196. Thesystem of claim 195, wherein the illuminator is coupled to thecontroller and configured to be turned on/off based on a control signalfrom the controller.
 197. The system of claim 195, wherein theilluminator is coupled to the controller through a memory bufferconfigured to control amplitude or brightness of one or more lightemitting elements in the illuminator.
 198. The system of claim 197,wherein the memory buffer for the illuminator has a smaller size than amemory buffer for the display.
 199. The system of claim 197, wherein anumber of the light emitting elements in the illuminator is smaller thana number of the elements of the display.
 200. The system of claim 195,wherein the controller is configured to simultaneously activate the oneor more light emitting elements of the illuminator.
 201. The system ofclaim 195, wherein the illuminator is a coherent light source, asemi-coherent light source, or an incoherent light source.
 202. Thesystem of claim 195, wherein the illuminator comprises two or more lightemitting elements each configured to emit light with a different color.203. The system of claim 195, wherein: the controller is configured tosequentially modulate the display with information associated with afirst color during a first time period and modulate the display withinformation associated with a second color during a second, sequentialtime period; and the controller is configured to control the illuminatorto sequentially turn on a first light emitting element to emit lightwith the first color during the first time period and a second lightemitting element to emit light with the second color during the secondtime period.
 204. The system of claim 195, wherein the illuminator isconfigured to emit a white light, and wherein the display is configuredto diffract the white light into light with different colors.
 205. Thesystem of claim 195, wherein the illuminator is arranged in front of asurface of the display and configured to emit the light on the surfaceof the display with an incident angle within a range between 0 degreeand 90 degree, and the emitted light is reflected from the surface ofthe display.
 206. The system of claim 205, wherein the emitted lightfrom the illuminator comprises collimated light.
 207. The system ofclaim 205, wherein the emitted light from the illuminator comprisesdivergent light.
 208. The system of claim 205, wherein the emitted lightform the illuminator comprises semi-collimated light.
 209. The system ofclaim 195, wherein: the illuminator is arranged behind a rear surface ofthe display and configured to emit a divergent light on the rear surfaceof the display; and the emitted light is transmitted through the displayand out of the display from a front surface of the display.
 210. Thesystem of claim 195, wherein the illuminator comprises: a light sourceconfigured to emit the light; and a waveguide coupled to the lightsource and arranged adjacent to the display, the waveguide beingconfigured to receive the emitted light from the light source and guidethe emitted light to the display.
 211. The system of claim 210, whereinthe light from the light source is coupled to the waveguide from a sidecross-section of the waveguide through a light coupler.
 212. The systemof claim 210, wherein the light source and the waveguide are integratedin a planar form and positioned on a surface of the display.
 213. Thesystem of claim 210, wherein the waveguide is configured to guide thelight to illuminate the display uniformly.
 214. The system of claim 210,wherein: the waveguide is positioned on a rear surface of the display,and the light is guided to transmit through the display and diffractedout of the display from a front surface of the display; and thecontroller is positioned on a rear surface of the waveguide.
 215. Thesystem of claim 210, wherein the waveguide is positioned on a frontsurface of the display, and wherein the light is guided to be incidenton the front surface of the display and reflected by the front surface.